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EXECUTIVE  SUMMARY 


This  report  describes  the  Demonstration  of  the  National  Convective  Weather  Forecast  (NCWF) 
Product  conducted  by  ACT-320  at  Delta,  Atlantic  Coast,  and  Northwest  Airlines’  dispatch 
offices.  The  Demonstration  was  conducted  between  June  20  and  October  19,  1998.  Specific 
results,  conclusions,  and  recommendations  for  the  Demonstration  are  detailed  within  the  report. 

The  NCWF  Product,  developed  by  the  National  Center  for  Atmospheric  Research  (NCAR),  is  a 
web-based  graphical  representation  of  current  and  forecast  (to  1  hour)  convectively  active 
regions,  both  nationally  and  regionally.  The  first  iteration  of  the  NCWF,  formerly  named  the 
Automated  Convective  SIGMET  Forecast  (ACSF)  Product,  was  demonstrated  in  1997  to  the 
Aviation  Weather  Center’s  (AWC)  Convective  SIGMET  Forecasters.  Although  the  Product  had 
merit,  findings  indicated  it  may  be  better  suited  to  nonmeteorological  aviation  weather  users, 
specifically  airline  dispatchers. 

Participants  in  the  NCWF  Demonstration  consisted  of  dispatchers  from  two  major  airlines,  Delta 
and  Northwest;  and  one  regional  airline,  Atlantic  Coast.  Demonstrations  were  conducted  on¬ 
site  at  each  of  the  airline’s  operational  dispatch  centers.  Objectives  of  the  Demonstration  were 
to  evaluate  the  extent  the  NCWF  benefited  dispatchers  in  performing  job  tasks  when  convective 
weather  effectively  impacted  the  airline’s  flight  routes.  Human-computer  interface  aspects  of  the 
Product,  as  well  as  performance  and  utility,  were  also  assessed.  Data  collection  metrics 
employed  included  on-line  user  logs,  questionnaires,  and  individual  structured  interview 
sessions. 

Overall  findings  from  all  airlines  were  generally  consistent.  With  few  exceptions,  the  NCWF  was 
rated  positively.  However,  issues  concerning  the  NCWF’s  performance,  as  well  as 
improvements,  were  also  identified.  These  included: 

1 .  The  forecast  component  was  sometimes  inaccurate,  especially  during  airmass  type  storms. 

2.  A  user-selected  zoom  capability  would  be  preferable  to  the  static  national  and  regional  map 
views. 

3.  A  longer  forecast  duration  would  help  meet  dispatch  flight  planning  requirements.  Usually 
flight  plans  are  issued  1  to  1  1/2  hours  prior  to  departure. 

4.  Display  animation  would  contribute  more  information  on  weather  trends. 

5.  A  growth  and  decay  component  would  give  a  better  general  picture  of  weather  dissipation 
as  well  as  intensity  for  dispatch  planning  purposes. 

6.  User-selected  overlays  such  as  major  cities,  airports,  VORs,  and  jet  routes  would  help 
narrow  the  focus  in  determining  locations  of  the  weather. 

Despite  issues  and  desired  improvements,  results  from  the  Demonstration  indicate  the  NCWF 
Product  has  merit  for  airline  dispatch  use  and  should  be  considered  for  future  development  and 
evaluation. 
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1.  INTRODUCTION. 


Thunderstorms,  or  convection,  account  for  nearly  half  of  national  airspace  delays  during  the 
summer  season.  Thunderstorm  phenomena  such  as  lightning,  turbulence,  hail,  rain,  icing,  and 
poor  visibility  require  airspace  users  to  provide  a  wide  margin  of  separation  between  these 
phenomena  and  aircraft  operations.  In  order  to  help  circumvent  these  phenomena  as  safely 
and  expeditiously  as  possible,  reliable  detection  and  forecasts  of  thunderstorms  and  convective 
activity  are  required. 

In  response  to  this  requirement,  prototype  convective  weather  detection  and  forecast  products 
are  being  developed  and  evaluated.  Under  the  Federal  Aviation  Administration’s  (FAA)  Aviation 
Weather  Research  (AWR)  Program,  the  following  organizations:  Massachusetts  Institute  of 
Technology’s  Lincoln  Laboratory  (MIT/LL);  the  National  Severe  Storms  Laboratory  (NSSL); 
National  Weather  Service  (NWS);  Aviation  Weather  Center  (AWC);  and  the  National  Center  for 
Atmospheric  Research  (NCAR)  are  collaborating  on  developing  automated  techniques  for  short¬ 
term  (0-3  hours)  forecasts  of  convection  for  the  aviation  community.  As  part  of  this  collaborative 
effort,  the  National  Convective  Weather  Forecast  (NCWF)  Product,  a  0-1  hour  forecast  product 
developed  by  NCAR,  was  evaluated  by  the  FAA  to  ascertain  whether  or  not  it  meets  the  aviation 
community’s  requirements  for  a  graphical,  convective,  forecast  and  detection  product.  The 
web-based  NCWF  Product  provides  graphical  information  regarding  the  current  (detection)  and 
forecast  locations  of  thunderstorms  both  nationally  and  regionally.  The  algorithm  produces  a 
60-minute  forecast. 

The  NCWF  Product  was  first  assessed  during  a  demonstration  at  the  AWC  during  the  summer 
of  1997  as  the  Automated  Convective  SIGMET  Forecast  (ACSF)  Product.  During  this 
demonstration,  Convective  SIGMET  forecasters  used  the  forecast  and  detection  product  as  a 
tool  to  aid  in  creating  Convective  SIGMETs.  The  assessment  was  conducted  by  personnel  from 
the  FAA  William  J.  Hughes  Technical  Center  and  NCAR.  The  demonstration  revealed  that 
some  components  of  the  Product  had  merit;  however,  specific  improvements  would  be  required 
in  order  to  make  it  useful  for  AWC  Convective  SIGMET  generation.  Results  from  this 
demonstration  are  documented  in  the  Technical  Center’s  Automated  Convective  SIGMET 
Forecast  Product  Demonstration  Final  Report,  December  1 997. 

Using  feedback  from  Convective  SIGMET  forecasters,  NCAR  developers  modified  the  Product 
display  and  worked  on  improving  the  Product  algorithms.  For  the  revised  NCWF  Product, 

NCAR  placed  an  increased  emphasis  on  other  potential  aviation  users,  particularly  airline 
dispatchers.  To  address  the  needs  of  both  small  and  large  airlines,  regional  and  major  airlines 
were  asked  to  participate  in  the  demonstration. 

To  assess  the  perceived  performance  and  value  of  the  NCWF  Product  for  airline  dispatch  use, 
the  Technical  Center  and  NCAR  conducted  an  NCWF  Product  demonstration  throughout  the 
summer  and  early  fall  of  1998.  Two  major  and  one  regional  airline  participated  in  this 
demonstration.  These  included  (1)  Northwest  Airlines,  (2)  Delta  Airlines,  and  (3)  Atlantic  Coast 
Airlines.  Demonstrations  were  conducted  at  each  airline’s  dispatch  operations’  center  located  in 
Minneapolis,  MN;  Atlanta,  GA;  and  Dulles,  VA;  respectively. 
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1.1  PURPOSE  OF  REPORT. 

The  purpose  of  this  report  is  to  document  feedback  and  data  collected  as  well  as  discussions 
and  conclusions  addressing  the  extent  to  which  the  NCWF  Product  met  stated  demonstration 
objectives. 


1.2  SCOPE  OF  REPORT. 

The  NCWF  Demonstration  Report  will  summarize  demonstration  conduct  and  results  regarding 
Product  utility,  user  interface,  and  overall  perceived  benefit  of  the  NCWF  Product  for  airline 
dispatchers.  A  separate  report,  prepared  by  NCAR,  will  address  the  scientific  and 
meteorological  aspects  of  the  NCWF  Demonstration. 


2.  REFERENCE  DOCUMENTS. 

FAA-STD-024B  U.  S  Department  of  Transportation,  Federal  Aviation  Administration 

Standard,  Content  and  Format  Requirements  for  the  Preparation  of  Test 
and  Evaluation  Documentation. 

Acquisition  Management  System,  Test  and  Evaluation  Process 
Guidelines,  February  1999. 

National  Convective  Weather  Forecast  (NCWF)  Product  Demonstration 
Plan  and  Procedures,  June  1998. 

Automated  Convective  SIGMET  Forecast  (ACSF)  Product  Demonstration 
Final  Report,  December  1997. 


3.  SYSTEM  DESCRIPTION. 

The  NCWF  Product  determines  the  locations  of  thunderstorm  hazards  from  the  integration  of 
lightning  and  radar  data.  The  lightning  data  is  mapped  to  a  Cartesian  grid  by  assigning  each 
lightning  strike  a  circle  radius  of  two  grids  covering  a  16  kilometer  (km)  diameter  circle. 

Lightning  rate  is  calculated  over  a  5-minute  period.  National  Scale  Vertically  Integrated  Liquid 
(VIL)  is  compared  to  corresponding  EchoTop  data  and  removed  if  the  echo  top  value  is  less 
than  17,000  feet.  This  technique  removes  the  majority  of  Anomalous  Propagation  (AP).  The 
VIL  and  lightning  rate  fields  are  then  combined  to  create  the  detection  field. 

The  VIL  data  field  is  also  used  in  the  forecast.  This  field  is  filtered  using  a  scale  separation  filter 
that  eliminates  small-scale  features  (those  perishable  within  an  hour)  providing  a  more  accurate 
forecast  of  long  lived  linear  storm  systems.  The  Thunderstorm  Identification  Tracking  and 
Nowcasting  (TITAN)  software  is  used  both  to  determine  storm  motion  from  the  filtered  data  field 
and  to  identify  storms  in  the  detection  field.  Storms  that  have  a  projected  area  of  520  km2  and 
are  over  22,000  feet  in  altitude  are  extrapolated.  Using  values  from  the  extrapolated  storm  field, 
a  storm  motion  vector  is  assigned  to  each  of  the  detected  storms. 
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The  data  sources  used  for  the  NCWF  algorithm  are  the  WSI  WSR88D  VIL,  Echo  Tops  Mosaics, 
and  National  Lightning  Detection  Network  (NDLN)  data  from  Global,  Inc.,  via  Kavouras.  The 
lightning  fields  are  4  km  in  resolution  and  updated  automatically  every  10  minutes.  Figure  1 
(AWC  Algorithms)  schematically  depicts  all  the  processes  (algorithms)  and  data  used  in 
creating  the  NCWF  Product  output.  Meteorological  Data  Volume  (MDV)  indicates  gridded  data 
whereas  SYMPROD  database  (SPDB)  represents  text  files.  MDV  and  SPDB  are  both 
acronyms  used  internally  by  NCAR. 


3.1  NCWF  PRODUCT  COMPONENTS. 

Figure  2  illustrates  one  of  the  regional  views  (Great  Lakes)  of  the  NCWF  Product.  Various 
components  of  the  NCWF  Product  and  display  are  defined  below. 

3.1.1  Initial  Detection  Field. 

The  initial  detection  field  (figure  2,  item  1)  displays  current  convective  activity,  and  is  depicted  by 
a  6-level  intensity  color  scale  with  highest  intensity  levels  (5-6)  as  red  and  lowest  intensity  levels 
(1-2)  as  green.  When  convective  detection  criteria  are  met,  the  colored  field  will  be  displayed, 
indicating  areas  most  likely  to  contain  convective  activity.  Although  Kavouras  provided  radar  is 
one  of  the  components  of  the  Initial  Detection  Field  data  ingests,  this  product  should  not  be 
confused  with  commercial  radar  output  because  lightning  data  and  filtering  of  low  echo  top 
regions  are  used  in  its  creation.  Additionally,  VIL  radar  output  is  not  commonly  used  in  aviation 
centers. 

3.1 .2  Convective  Intensity  Color  Scale. 

The  convective  intensity  color  scale,  (figure  2,  item  2)  located  on  the  right  of  the  display, 
indicates  intensity  levels  of  convective  activity  in  a  similar  manner  commonly  seen  on  radar 
displays,  i.e.,  red  indicating  the  highest  intensity  and  green  the  lowest. 

3.1.3  Detection  Field  Colors. 

Detection  field  colors  (figure  2,  item  1)  are  the  6-level  intensity  colors  indicated  by  the  color 
intensity  scale. 

3. 1 .4  Forecast  Contours. 

Forecast  contours  (figure  2,  item  3)  are  cyan  colored  polygons  indicating  areas  where 
convective  activity  may  occur  in  1  hour.  The  forecast  option  may  be  toggled  on  or  off  by 
selecting  the  forecast  on/off  button  on  the  bottom  left  of  the  planview  display. 

3.1.5  Storm  Height  Information. 

Storm  height  information  (figure  2,  item  4)  is  numerically  annotated  in  hundreds  of  feet  and 
located  over  the  displayed  detection  and  forecast  convective  areas. 
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3.1 .6  Echo  Speed  Values. 


Echo  speed  values  (figure  2,  item  5)  are  numerically  annotated  in  knots  per  hour  along  with 
storm  height  information. 

3.1 .7  Forecast  Direction  Lines. 

Forecast  direction  lines  (figure  2,  item  6)  are  displayed  as  white  lines  or  vectors,  indicating  the 
direction  of  storm  movement. 

3.1.8  Views. 

Geographical  areas  of  interest  are  divided  by  “views.”  Users  may  select  a  national  view  to 
observe  the  contiguous  United  States,  or  smaller  views  such  as  the  Eastern  United  States, 
Southwest,  or  a  specific  Air  Route  Traffic  Control  Center  (ARTCC)  area.  An  example  of  the 
Great  Lakes  regional  view  is  shown  in  figure  2. 
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AWC  ALGORITHMS 
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FIGURE  1.  AWC  ALGORITHMS 


Local  Time  GMT  Time 


FIGURE  2.  NATIONAL  CONVECTIVE  WEATHER  FORECAST  PRODUCT 


4.  DEMONSTRATION  DESCRIPTION. 


4.1  DEMONSTRATION  SCHEDULE  AND  LOCATIONS. 

The  NCWF  operational  demonstration  took  place  at  the  following  airline  dispatch  operations 
centers: 


a.  Delta  Airlines  Operations  Center,  Atlanta,  GA; 

b.  Northwest  Airlines  Systems  Operations  Center  (SOC),  Minneapolis,  MN;  and 

c.  Atlantic  Coast  Airlines  Dispatch  Operations,  Sterling,  VA  (Washington-Dulles  Airport). 
The  demonstration  schedule  for  each  airline  was  as  follows: 

a.  Delta  Airlines:  June  22  through  July  28,  1 998; 

b.  Northwest  Airlines:  June  22  through  October  19, 1998.  (Note:  The  Northwest 
demonstration  phase  was  extended  to  accommodate  the  pilot  strike  and  month-long  interruption 
in  operations.); 

c.  Atlantic  Coast  Airlines:  July  20  through  September  7, 1 998. 


4.2  PARTICIPANTS. 

Demonstration  participants  at  each  airline  consisted  of  the  following: 

a.  Delta  Airlines:  flight  superintendents  (dispatchers)  and  dispatch  Air  Traffic  Control 
(ATC)  liaisons; 

b.  Northwest  Airlines:  dispatchers,  dispatch  supervisors,  and  dispatch  liaisons; 

c.  Atlantic  Coast  Airlines:  dispatchers,  dispatch  team  leads,  and  dispatch  supervisors. 

Because  all  categories  of  dispatchers  are  involved  in  dispatch  operations  and  the  overall 
participant  response  was  relatively  small,  all  participants  have  been  categorized  as 
“dispatchers.” 


4.3  DEMONSTRATION  AND  SPECIALIZED  EQUIPMENT. 

No  specialized  equipment  was  required.  Since  NCWF  is  a  web-based  product,  Internet 
capability  on  computer  display  monitors  were  the  only  requirements  for  Product  access. 
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4.4  DEMONSTRATION  OBJECTIVES/CRITERIA. 


The  objectives  of  the  NCWF  Product  Demonstration  were  to  determine  the  following: 

a.  Does  the  NCWF  Product  add  value  to  identifying  and  forecasting  areas  of  convective 
weather  activity? 

b.  Does  the  NCWF  Product  benefit  tasks  and  decisions  affected  by  convective  weather 
activity? 

c.  How  well  does  the  Product  perform?  Under  what  situations  does  the  Product  perform 
well  or  not  perform  well? 

d.  How  can  the  Product  be  enhanced?  What  improvements  or  changes  would  this 
necessitate? 

e.  Is  a  1  -hour  forecast  Product  adequate  for  airline  dispatchers’  needs? 

f.  Are  the  NCWF  Product  interfaces  and  characteristics  operationally  acceptable? 

The  extent  to  which  these  objectives  have  been  met  is  discussed  in  sections  5  and  6  of  this 
report. 

4.5  DEMONSTRATION  DESCRIPTIONS. 

The  NCWF  Demonstration  was  conducted  in  two  phases.  Activities  during  phase  1  consisted  of 
airline  coordination,  demonstration  planning,  initial  Product  concept  feedback,  and  training. 
Phase  2  included  actual  demonstration  conduct  where  dispatchers  were  asked  to  look  at  and/or 
use  the  Product  operationally  and  provide  feedback  in  the  form  of  daily  logs,  questionnaires, 
and  structured  interviews.  Each  of  these  phases  is  discussed  in  detail  in  subsequent 
paragraphs. 

4.5.1  Phase  1  Demonstration  Activities. 

4.5.1 .1  Product  Concept  Development. 

Three  potential  NCWF  Product  options  were  informally  briefed  to  dispatchers  in  order  to  obtain 
feedback  on  a  preferred  Product  presentation;  i.e.,  placement  of  locators,  incorporation  of  range 
rings,  and  VOR  overlays.  Based  on  these  responses,  NCAR  created  a  final  Product  concept  for 
demonstration  purposes.  Figure  1  illustrates  the  final  NCWF  Product. 

4.5.1 .2  Training. 

Training  was  conducted  at  each  of  the  airlines’  operational  dispatch  environments.  Training, 
provided  by  NCAR  and  monitored  by  ACT-320  test  personnel,  was  hands-on,  individual,  and 
interactive.  A  one-page  quick  reference  guide  was  provided  to  all  trainees  explaining  Product 
components,  limitations,  and  contacts  for  troubleshooting.  Due  to  time  and  resource 
constraints,  all  dispatchers  were  not  trained.  However,  airlines  agreed  to  allow  previously 


trained  dispatchers  to  familiarize  other  dispatchers  on  their  shifts.  Since  Product  use  is  fairly 
intuitive;  the  interface  is  fundamentally  simple;  and  displayed  data  is  understandable;  extensive 
training  was  not  considered  an  issue. 

4.5.2  Phase  2  Demonstration  Activities. 

4.5.2.3  Questionnaires. 

At  the  end  of  the  demonstration  phase,  ACT-320  test  personnel  distributed  questionnaires  to  all 
dispatchers  at  each  of  the  three  airlines.  The  questionnaire  contained  two  sections. 

4.5.2.3.1  Section  1. 

In  section  1,  dispatchers  were  asked  to  rate  the  NCWF  Product  and  components  on  the 
dimensions  of  utility,  reliability,  confidence,  and  readability  using  a  5-point  Likert  scale.  Product 
components  and  functions  were  rated  on  all  or  a  subset  of  these  dimensions.  Items  rated 
included: 

a.  Initial  Detection  Fields 

b.  Convective  Intensity  Color  Scale 

c.  Detection  Field  Colors 

d.  Forecast  Contours 

e.  Forecast  Direction  Lines 

f.  50  nautical  miles  (nmi)  Range  Rings 

g.  Storm  Height  Information 

h.  Echo  Speed  Values 

i.  ARTCC  Boundary  Overlays 

j.  ARTCC  Map 

k.  Views:  National,  Eastern,  Southern,  et  al 

l.  Regional  Views:  Northeast,  Great  Lakes,  et  al 

m.  Navigation  Between  Views 

n.  Product  Update  Rate 

Rating  anchors  included  the  following: 

a.  Largely  Acceptable  -  This  response  indicates  the  Product/component  being  assessed 
consistently  enhances  your  ability  to  meet  the  requirements  of  your  job;  likely  to  lead  to 
enhanced  job  performance. 

b.  Acceptable  -  This  response  indicates  the  Product/component  being  assessed 
frequently  enhances  your  ability  to  meet  the  requirements  of  your  job;  may  lead  to  enhanced  job 
performance. 

c.  Borderline  -  This  response  indicates  that,  although  the  Product/component  being 
assessed  is  adequate,  minor  improvements  would  make  it  more  helpful  in  identifying  and 
forecasting  convective  weather  and  does  not  degrade  or  improve  job  performance. 

d.  Unacceptable  -  This  response  indicates  the  Product/component  being  assessed 
frequently  impedes  your  ability  to  meet  the  requirements  of  your  job;  may  lead  to  degradation  of 
job  performance. 
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e.  Largely  Unacceptable  -  This  response  indicates  the  Product/component  being 
assessed  consistently  impedes  your  ability  to  meet  the  requirements  of  your  job;  likely  to  lead  to 
degradation  of  job  performance. 

4.5.2.3.2  Section  2. 

The  second  section  assessed  Product  usefulness  in  relation  to  operational  tasking.  Tasks  were 
identified  during  the  first  phase  of  the  demonstration  and  verified  by  dispatch  management. 
Again,  a  5-point  Likert  scale  was  used  and  Product  usefulness  was  measured  against  specific 
tasks.  Questions  relating  Product  usefulness  to  dispatcher  tasking  included  the  following: 

a.  Preparing  flight  plans 

b.  Planning  fuel  quantity 

c.  Planning  flight  route 

d.  Avoidance  of  severe  convective  weather 

e.  Selecting  an  alternate  airport 

f.  Selecting  altitude 

g.  Monitoring  flight  conditions 

h.  Rerouting  flights 

i.  Advising  pilots  enroute  of  severe  convective  weather 

j.  Anticipating  ground  delays 

k.  Anticipating  ATC  actions  (i.e.,  rerouting,  closing  runways,  changing  arrival  gates) 

Rating  anchors  ranged  from  1,  “Of  Considerable  Use”  to  5,  “Not  at  all  Useful.”  A  rating  of  3 
indicated  the  Product  had  little  to  no  effect  either  positively  or  negatively,  on  decisions  and 
activities  related  to  the  performance  of  the  task.  Rating  anchors  included  the  following: 

a.  Of  Considerable  Use.  This  response  indicates  the  NCWF  Product  had  a  significant 
positive  effect  on  decisions  and  activities  related  to  the  performance  of  this  task. 

b.  Somewhat  Useful.  This  response  indicates  the  NCWF  Product  had  a  positive  effect  on 
decisions  and  activities  related  to  the  performance  of  the  task. 

c.  Borderline.  This  response  indicates  the  NCWF  Product  had  little  to  no  effect  on 
decisions  and  activities  related  to  the  performance  of  the  task. 

d.  Of  Little  Use.  This  response  indicates  the  NCWF  had  a  negative  effect  on  decisions  and 
activities  related  to  the  performance  of  the  task. 

e.  Not  at  all  Useful.  This  response  indicates  the  NCWF  Product  had  a  significant  negative 
effect  on  decisions  and  activities  related  to  the  performance  of  the  task. 

The  questionnaire  also  included  two  open-ended  questions  soliciting  suggestions  for 
improvement  and  additional  capabilities/components.  The  questionnaire  is  located  in  appendix 
A  and  a  compilation  of  questionnaire  comments  from  the  open-ended  questions  are  located  in 
appendix  B. 
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4.5. 2.4  Structured  Interviews. 


Concurrent  with  questionnaire  administration,  structured  interviews  were  conducted  individually 
within  the  dispatch  area  by  ACT-320  test  personnel.  The  intent  of  the  interview  was  to  solicit 
information  that  may  not  be  ratable  within  the  questionnaire  format  and  to  provide  more  detail. 
These  included  the  extent  of  Product  use,  how  it  compared  to  other  convective  weather 
sources,  value  added  to  operations,  instances  where  the  Product  performed  well  or  poorly,  and 
requirements  for  forecast  times.  Structured  interview  questions  and  summaries  are  located  in 
appendices  C  and  D. 


4.5.2.5  Daily  User  Log. 

Throughout  the  demonstration,  dispatchers  were  also  asked  to  provide  daily  comments  on  the 
user  feedback  form  linked  to  the  NCWF  Product  web  page.  Users  were  encouraged  to  note  any 
negative  or  positive  performance  aspects  of  the  Product  during  day-to-day  use.  A  sample  of  the 
user  log  form  is  located  in  appendix  E. 


4.6  DATA  COLLECTION  AND  ANALYSIS  METHOD. 

Descriptive  statistics  were  used  for  data  analysis.  Rating  scores  were  not  normally  distributed 
and  data  was  ordinal,  therefore,  the  median  was  used  as  a  measure  of  central  tendency.  One 
half  of  the  observations  fall  above  and  below  the  median  value.  When  there  is  an  even  number 
of  observations,  no  unique  center  value  exists,  so  the  mean  of  the  two  middle  observations  is 
taken  as  the  median  value.  Percentage  values  of  rating  scale  responses  were  also  calculated. 
Responses  from  questionnaire  comments,  and  structured  interview  questions  were 
summarized. 

4.6.1  Factors  Affecting  Results. 

a.  Small  Sample  Size  -  The  number  of  questionnaire  responses  from  all  participating 
airlines  was  poor.  At  Delta,  for  example,  only  1 6  questionnaires  were  completed  (out  of  the  145 
distributed  to  all  dispatchers).  At  Northwest,  nine  questionnaires  were  returned  and  Atlantic 
Coast  Airlines  completed  three.  Whether  these  samples  are  representative  of  Delta,  Northwest, 
and  Atlantic  Coast  Airlines;  or  airline  dispatchers  in  general,  is  questionable. 

b.  Lack  of  Operationally  Significant  Convective  Weather  -  Operationally  significant 
convective  weather  activity  was  not  prevalent  throughout  the  demonstration  period.  Although 
there  were  some  occurrences,  the  Product  may  not  have  had  a  chance  to  be  fully  utilized. 

c.  Traininq/Familiarization  Constraints  -  Training  was  conducted  individually  in  the 
operational  environment  at  each  participating  airline.  With  a  dispatch  population  of  over  140  at 
both  Delta  and  Northwest  Airlines,  resources  and  time  to  train  each  dispatcher  were 
unavailable.  Although  those  who  missed  training  were  to  be  trained  by  other  trained 
dispatchers,  this  was  not  always  the  case.  Consequently,  many  dispatchers  were  unaware  the 
NCWF  existed  or  were  too  unfamiliar  with  the  Product  to  use  it. 
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d.  Product  Inaccessibility  -  The  ability  of  dispatchers  to  use  the  NCWF  Product  at  Atlantic 
Coast  Airlines  was  limited  since  the  Product  was  only  displayed  at  one  workstation  and  not 
viewable  from  other  workstations.  In  order  to  view  the  display,  most  dispatchers  had  to  walk 
across  the  room  and  around  the  workstation.  This  was  both  time  consuming  and  disruptive, 
accounting  for  limited  operational  use  of  the  Product. 

e.  Northwest  Airlines  Pilot  Strike  -  The  pilot  strike  occurred  during  the  middle  of  the 
demonstration  period.  Until  its  resolution,  all  flight  operations  ceased  for  approximately 

1  month.  Although  the  NCWF  demonstration  continued  from  that  point,  the  interruption  may 
have  compromised  user  motivation  in  using  the  Product  and  providing  feedback. 

5.  RESULTS. 

5.1  QUESTIONNAIRE  RESULTS. 

Overall  results  from  questionnaire  rating  and  comment  questions  for  all  airline  dispatcher 
participants  are  discussed  below.  Additionally,  individual  results  will  be  discussed  based  on 
participant  responses  from  each  of  the  three  airline  dispatch  groups. 


5.1.1  Overall  Results. 

5.1 .1 .1  Product  Components  and  Interfaces. 

Overall  median  scores  from  data  from  all  participating  airline  dispatchers  were  derived.  Table  1 
represents  overall  acceptability  ratings  from  the  first  section  of  the  questionnaire  (questions  1  — 
14)  regarding  the  human  interface/  Product  component  questions. 

In  the  first  section  of  the  questionnaire,  all  Product  components  and  interfaces  were  rated 
positively,  receiving  an  overall  rating  of  2  (acceptable)  on  all  dimensions  rated  including  utility, 
reliability,  readability,  and  confidence.  (Note:  Not  all  interfaces  and  components  were  rated  on 
all  dimensions.) 

5.1.1 .2  Product  Improvements. 

Product  improvement  questionnaire  comments  are  addressed  under  individual  airline  headings. 
An  overall  summary  of  improvements  is  listed  in  appendix  B. 

5.1. 1.3  Product  Usefulness. 


Median  rating  scores  from  the  second  part  of  the  questionnaire  (questions  17  -  27)  assessing 
Product  usefulness  were  also  positive  overall.  NCWF  usefulness  was  rated  highest  (1  —  of 
considerable  use)  for  avoiding  convective  weather  and  advising  pilots  of  convective  weather 
activity.  Product  usefulness  for  flight  monitoring  and  planning  flight  routes  also  received  positive 
ratings  with  a  median  score  of  1 .5.  A  median  score  of  2  was  obtained  for  fuel  planning, 
rerouting,  and  anticipating  ATC  decisions,  indicating  the  NCWF  Product  was  useful.  Product 
usefulness  for  selecting  altitude,  anticipating  ground  delays,  and  anticipating  ATC  actions 
received  a  borderline  rating  of  3,  indicating  the  NCWF  Product  had  little  to  no  effect  on 
decisions  and  activities  related  to  the  performance  of  the  tasks.  See  table  2  for  overall  Product 
usefulness  ratings. 
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TABLE  1.  OVERALL  ACCEPTABILITY  MEDIAN  RATINGS  AND  PERCENTAGES  OF 
PRODUCT  COMPONENTS  AND  INTERFACES  FOR  DELTA,  NORTHWEST,  AND  ATLANTIC 

COAST  AIRLINES 


COMPONENT/FEATURE 

■ 

IjjjjH 

Acceptable 

Borderline 

Unacceptable 

Largely 

Unacceptable 

1 

2 

3 

4 

5 

Total 

N 

N 

N 

N 

N 

Mdn 

N 

% 

% 

% 

% 

% 

1.  Initial  Detection  Fields 

a.  Utility 

27 

6 

17 

4 

0 

0 

2 

22.2 

63 

14.8 

0 

0 

b.  Reliability 

27 

7 

12 

3 

1 

0 

30.4 

52.2 

13 

4.3 

0 

c.  Readability 

27 

6 

19 

2 

0 

0 

2 

22.2 

70.4 

7.4 

0 

0 

d.  Confidence 

27 

6 

15 

5 

1 

0 

2 

22.2 

55.6 

18.5 

3.7 

0 

2.  Convective  Intensity 

Color  Scale 

a.  Utility 

28 

12 

14 

2 

0 

0 

2 

42.9 

50.0 

7.1 

0 

0 

b.  Reliability 

26 

10 

13 

2 

1 

0 

2 

38.5 

50 

7.7 

3.8 

0 

c.  Readability 

28 

11 

15 

2 

0 

0 

2 

39.3 

53.6 

7.1 

0 

0 

3.  Detection  Field  colors 

a.  Utility 

27 

9 

15 

2 

0 

1 

2 

32.1 

53.6 

7.1 

0 

3.7 

b.  Readability 

27 

9 

16 

1 

0 

1 

2 

33.3 

59.3 

3.7 

0 

3.7 

4.  Forecast  Contours 

a.  Utility 

28 

9 

17 

2 

0 

0 

2 

32.1 

60.7 

7.1 

0 

0 

b.  Reliability 

27 

8 

15 

3 

1 

0 

2 

29.6 

55.6 

11.1 

3.7 

0 

c.  Readability 

28 

7 

19 

2 

0 

0 

2 

25 

67.9 

7.1 

0 

0 

d.  Confidence 

28 

8 

12 

7 

1 

0 

2 

28.6 

42.9 

25 

3.6 

0 

5.  Forecast  Direction 

Lines 

a.  Utility 

26 

7 

14 

5 

0 

0 

2 

26.9 

53.8 

19.2 

0 

0 

b.  Reliability 

25 

6 

15 

2 

1 

1 

2 

24 

60 

8 

4 

4 

c.  Readability 

26 

4 

18 

4 

0 

0 

2 

15.4 

69.2 

14.3 

0 

d.  Confidence 

26 

3 

16 

7 

0 

111 

11.5 

61.5 

26.9 

0 

■ 

TABLE  1 .  OVERALL  ACCEPTABILITY  MEDIAN  RATINGS  AND  PERCENTAGES  OF 
PRODUCT  COMPONENTS  AND  INTERFACES  FOR  DELTA,  NORTHWEST,  AND  ATLANTIC 

COAST  AIRLINES  (CONTINUED) 


COMPONENT/FEATURE 

m 

IBB 

Acceptable 

Borderline 

Unacceptable 

Largely 

Unacceptable 

■ 

i 

2 

3 

4 

5 

Total 

N 

N 

N 

N 

N 

Mdn 

N 

% 

% 

% 

% 

% 

6.  50  nm  Range  Rings 

a.  Utility 

23 

10 

12 

1 

0 

0 

2 

43.5 

52.2 

4.3 

0 

0 

b.  Readability 

22 

11 

12 

0 

0 

0 

2 

47.8 

52.2 

0 

0 

0 

7.  Storm  Height 

Information 

a.  Utility 

28 

9 

16 

3 

0 

0 

2 

32.1 

57.1 

10.7 

0 

0 

b.  Reliability 

mam 

16 

2 

1 

0 

2 

61.5 

7.7 

3.8 

0 

c.  Readability 

12 

6 

0 

0 

2 

44.4 

22.2 

0 

0 

28 

5 

19 

3 

1 

0 

2 

HHHjjfljH 

17.9 

67.9 

10.7 

3.6 

0 

28 

6 

20 

2 

0 

0 

2 

21.4 

71.4 

7.1 

0 

0 

26 

6 

17 

2 

1 

0 

2 

■■■Mi 

23.1 

65.4 

7.7 

3.8 

0 

18 

5 

19 

4 

0 

0 

2 

17.9 

67.9 

14.3 

0 

0 

d.  Confidence 

28 

5 

19 

3 

1 

0 

2 

17.9 

67.9 

10.7 

3.6 

0 

9.  ARTCC  Boundary 

Overlays 

a.  Utility 

26 

12 

12 

1 

1 

0 

2 

46.2 

46.2 

3.8 

3.8 

0 

b.  Readability 

25 

12 

12 

0 

1 

0 

2 

48 

48 

0 

4 

0 

10.  ARTCC  Map 

a.  Utility 

26 

12 

12 

1 

0 

1 

2 

46.2 

46.2 

3.8 

0 

3.8 

b.  Readability 

26 

12 

13 

0 

1 

0 

2 

46.2 

50 

0 

3.8 

0 

11.  Views  (National, 

Eastern,  et  al) 

a.  Utility 

28 

6 

18 

3 

0 

1 

2 

21.4 

64.3 

10.7 

0 

3.6 

b.  Readability 

28 

9 

14 

5 

0 

0 

2 

32.1 

50 

17.9 

0 

0 

_j 
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TABLE  1 .  OVERALL  ACCEPTABILITY  MEDIAN  RATINGS  AND  PERCENTAGES  OF 
PRODUCT  COMPONENTS  AND  INTERFACES  FOR  DELTA,  NORTHWEST,  AND  ATLANTIC 

COAST  AIRLINES  (CONTINUED) 


COMPONENT/FEATURE 

■ 

Largely 

Acceptable 

Acceptable 

Borderline 

Unacceptable 

Largely 

Unacceptable 

1 

2 

3 

4 

5 

Total 

N 

N 

N 

N 

N 

mm 

N 

% 

% 

% 

% 

% 

12.  Regional  views 

a.  Utility 

28 

6 

0 

2 

21.4 

0 

b.  Readability 

28 

7 

1 

2 

25 

3.6 

0 

13.  Navigation  Between 

26 

5 

1 

1 

2 

Views 

19.2 

3.8 

3.8 

14.  Product  Update  Rate 

27 

9 

■ 

1 

2 

33.3 

3.7 

15  and  16:  See  Product 

Improvement  summaries  in 

Appendix  B 
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TABLE  2.  OVERALL  USEFULNESS  MEDIAN  RATINGS  AND  PERCENTAGES  FOR 
DISPATCHER  TASKING  FOR  DELTA,  NORTHWEST  AND  ATLANTIC  COAST  AIRLINES 


TASK 

Of 

Considerable 

Use 

Somewhat 

Useful 

Borderline 

Of 

Little 

Use 

Not  at  all 
Useful 

■ 

1 

2 

3 

4 

5 

N 

N 

N 

N 

N 

Mdn 

% 

% 

% 

% 

% 

17.  Preparing  Flight  Plans 

28 

9 

14 

3 

2 

0 

2 

32.1 

50 

10.7 

7.1 

0 

18.  Planning  Fuel 

27 

5 

13 

3 

0 

2 

Quantity 

18.5 

48.1 

11.1 

0 

19.  Planning  Flight  Route 

28 

14 

10 

i  2 

2 

0 

1.5 

50 

35.7 

7.1 

7.1 

0 

20.  Avoidance  of  Severe 

28 

16 

10 

0 

1 

Convective  Weather 

57.1 

35.7 

0 

21 .  Selecting  an  Alternate 

28 

6 

10 

9 

3 

0 

2 

Airport 

21.4 

35.7 

32.1 

10.7 

22.  Selecting  Altitude 

28 

1 

12 

7 

8 

0 

3 

3.6 

42.9 

25 

28.6 

0 

23.  Monitoring  Flight 

28 

15 

9 

3 

1 

0 

Conditions 

53.6 

32.1 

10.7 

3.6 

0 

24.  Re-routing  Flights 

28 

14 

11 

3 

0 

0 

1.5 

1 

50 

39.3 

10.7 

0 

0 

25.  Advising  Pilots 

27 

17 

7 

2 

1 

0 

1 

Enroute  of  Severe 

63 

25 

7.1 

3.7 

0 

Convective  Weather 

26.  Anticipating  Ground 

26 

5 

7 

9 

4 

1 

3 

Delays 

17.9 

25 

32.1 

14.3 

3.6 

27.  Anticipating  ATC 

28 

1 

12 

7 

8 

0 

3 

Actions 

3.6 

42.9 

25 

28.6 

0 

5.1 .2  Delta  Airlines  Questionnaire  Results. 

5.1 .2.1  Product  Components  and  Interfaces. 

Questionnaire  results  from  Delta  airline  dispatchers  were  similar  to  overall  results. 

In  the  first  section  of  the  questionnaire  (questions  1-14),  all  Product  interface  components  of 
the  NCWF  were  rated  positively.  Product  components  rated  via  a  combination  or  subset  of 
utility,  reliability,  readability,  and  confidence  received  rating  scores  of  2  (acceptable)  or  higher. 
Utility  of  the  50-nmi  range  rings  and  Product  Update  Rate  was  rated  highest  with  an  overall 
rating  score  of  1  (largely  acceptable).  Table  3  illustrates  overall  questionnaire  results  per  airline. 
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5.1 .2.1  Product  Improvements. 

Generally,  respondent  comments  on  Product  improvements  or  suggestions  (see  questions  15 
and  16  on  questionnaire)  focused  on  user  defined  zoom  capability,  animation  or  looping 
capability  (to  see  trending),  and  longer  forecast  durations  (2  to  6  hours). 

Criticism  centered  on  difficulty  reading  tops  and  echo  speed  annotations,  and  a  lack  of 
adequate  cities  and  airport  overlays  for  easier  identification  -  especially  for  those  airports  or 
cities  that  serve  as  hubs  (Atlanta,  Dallas/Ft.  Worth,  Cincinnati)  and  destinations. 

Delta  questionnaire  comment  summaries  from  questions  15  and  16  are  located  in  appendix  B. 


5.1 .2.2  Product  Usefulness. 

In  the  Product  usefulness  section  of  the  questionnaire,  relating  the  Product  to  tasking,  overall 
rating  scores  were  positive.  With  the  exception  of  the  “Selecting  altitude”  task,  which  received 
an  overall  borderline  score  of  3,  all  other  categories  received  overall  ratings  of  2.5  and  above. 
Rating  scores  on  “Avoidance  of  severe  weather”  and  “Advising  pilots  enroute  of  severe 
convective  weather”  received  an  overall  rating  score  of  1  (“Of  Considerable  Use”).  A  bar  chart 
depicting  Delta’s  Product  usefulness  ratings  along  with  Northwest’s  and  Atlantic  Coast  Airline’s 
are  provided  in  figure  3. 
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TABLE  3.  MEDIAN  RANKINGS  ON  UTILITY,  READABILITY,  RELIABILITY,  AND 
CONFIDENCE  PER  COMPONENT/FEATURE  BY  INDIVIDUAL  AIRLINE  DISPATCHERS 


1.  Initial  Detection  Field 


a.  Utilit 


b.  Reliabilit 


c.  Readabilit 


d.  Confidence 


2.  Convective  Intensity  Color  Scale 


b.  Reliabilit 


c.  Readabili 


3.  Detection  Field  Colors 


a.  Utilit 


b.  Readabilit 


4.  Forecast  Contours _ 

a.  Utilit 


b.  Reliabilit 


c.  Readabilit 


d.  Confidence 


5.  Forecast  Direction  Lines 
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2 
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2 
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6.  50  nm  Range  Rings 


a.  Utili 


b.  Readability _ 

7.  Storm  Height  Information 


a.  Utili 


b.  Reliabilit 


c.  Readabilit 


d.  Confidence 


8.  Echo  Speed  Values _ 


a.  Utili 


b.  Reliabili 


c.  Readabili 


d.  Confidence 


9.  ARTCC  Boundary  Overlays 


a.  Utili 
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10.  ARTCC  Ma 


a.  Utili 
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TABLE  3.  MEDIAN  RANKINGS  ON  UTILITY,  READABILITY,  RELIABILITY,  AND 
CONFIDENCE  PER  COMPONENT/FEATURE  BY  INDIVIDUAL  AIRLINE  DISPATCHERS 

(CONTINUED) 


5.1 .3  Northwest  Airlines  Questionnaire  Results. 

Results  from  the  Northwest  questionnaire  were  similar  to  Delta’s.  In  the  first  section,  almost  all 
Product  interface  components  of  the  NCWF  were  rated  positively  receiving  rating  scores  of  2 
(acceptable)  or  higher.  Utility  and  readability  of  the  50-nmi  range  rings  received  the  highest 
positive  rating  score  of  1.5.  Acceptability  of  the  NCWF  Product  update  rate,  however,  received 
an  overall  borderline  rating  of  3. 


5.1. 3.1  Product  Improvements. 

Most  comments  addressed  the  need  for  additional  display  identifiers  and  overlays  for  jet  routes, 
navigational  aids  (navaids),  fixes,  cities,  VORs,  and  other  geographical  features  such  as  roads 
and  rail  lines.  One  suggested  overlaying  the  NCWF  Product  on  the  current  Aircraft  Situation 
Display  (ASD)  which  already  provides  this  information. 

User  selected  zoom  and  animation  capabilities  were  also  suggested  as  well  as  the  need  for  a 
longer  forecast  time  -  up  to  4  hours  for  flight  planning  purposes.  The  capability  to  filter  out 
selected  levels  on  the  6-level  color  intensity  scale;  i.e.,  allowing  the  user  to  filter  out  convective 
levels  of  1  -  3  and  displaying  levels  4  -  6,  was  also  suggested. 

Northwest  questionnaire  comment  summaries  on  Product  improvements  are  located  in 
appendix  B. 


5.1. 3.2  Product  Usefulness. 

Product  usefulness  was  rated  highest  (1)  for  two  tasks: 

a.  Monitoring  flight  conditions,  and 

b.  Advising  pilots  enroute  of  severe  convective  weather 

Five  other  task  areas  received  positive  usefulness  ratings  (somewhat  useful)  and  include: 

a.  Planning  flight  routes 

b.  Rerouting  flights 

c.  Avoidance  of  severe  convective  weather 

d.  Selecting  alternate  airports,  and 

e.  Selecting  altitudes 

Three  task  areas,  however,  received  borderline  usefulness  ratings,  indicating  the  Product  had 
neither  a  positive  nor  negative  effect  on  the  task.  These  included: 

a.  Planning  fuel  quantity 

b.  Anticipating  ground  delays,  and 
c  Anticipating  ATC  actions 

A  bar  chart  depicting  Northwest  Airlines  Product  usefulness  ratings  along  with  Delta’s  and 
Atlantic  Coast's  are  provided  in  figure  3. 
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5.1 .4  Atlantic  Coast  Airlines  Questionnaire  Results. 


Results  from  the  Atlantic  Coast  Airlines  (ACA)  questionnaire  were  mostly  positive.  However, 
given  the  small  sample  size  (N  =  3),  results  cannot  be  considered  significant. 

In  the  first  section,  almost  all  Product  components  and  interfaces  received  positive  rating  scores 
of  2  (acceptable)  or  higher.  Approximately  40  percent  of  these  were  rated  largely  acceptable  on 
all  Product  component  dimensions  scoring  1 .5  and  above.  Dimensions  of  the  components  rated 
highest  included: 

a.  Utility  and  readability  of  the  convective  intensity  color  scale 

b.  Utility  of  the  forecast  contours,  and 

c.  Utility  and  readability  of  the  ARTCC  maps 

Utility  and  readability  of  the  main  views;  i.e.,  National,  Eastern,  and  Southern  received 
borderline  ratings.  Acceptability  of  Product  navigation  was  also  considered  borderline. 


5.1 .4.1  Product  Improvements. 

Overall,  easier  navigation  between  Product  views  was  suggested.  Need  for  a  longer  forecast 
duration,  2  to  4  hours,  was  also  noted.  Product  improvement  summaries  are  located  in 
appendix  B. 


5.1 .4.2  Product  Usefulness. 

Product  usefulness  was  rated  highest  in  five  of  the  identified  task  areas.  These  are  listed 
below. 

a.  Planning  flight  routes 

b.  Avoidance  of  severe  convective  weather 

c.  Rerouting  flights 

d.  Advising  pilots  enroute  of  severe  convective  weather 

e.  Anticipating  ATC  actions. 

With  the  exception  of  the  borderline  rating  for  the  “selecting  altitude”  task,  all  other  tasks 
received  positive  usefulness  ratings  (2).  A  bar  chart  depicting  Atlantic  Coast  Airlines  Product 
usefulness  ratings  along  with  Delta’s  and  Northwest's  are  provided  in  figure  3. 


5.2  STRUCTURED  INTERVIEW  RESULTS. 

Structured  interview  questions  focused  on  Product  use;  comparisons  to  other  convective 
weather  information  sources;  perceived  value  added  to  operations  effected  by  convective 
weather;  usefulness  of  Product  components;  and  scenarios  where  the  Product  did  or  did  not 
perform  well.  Comments  on  Product  enhancements  concerning  forecast  time  requirements  and 
animation  capabilities  were  also  solicited.  Additional  comments  on  Product  improvements  are 
also  included  in  these  summaries  and  may  be  referenced  in  appendix  C. 
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5.2.1  Delta  Airline  Interview  Results. 


ACT-320  conducted  1 7  individual  interviews  at  Delta’s  Operational  Control  Center  in  the 
operational  dispatch  area.  Results  are  summarized  according  to  interview  question. 

1.  Did  you  use  the  NCWF  Product? 

Most  respondents  (15)  had  used  the  NCWF  Product  to  some  extent.  Factors  affecting  why  the 
Product  was  not  used  extensively  included  comparatively  few  instances  of  severe  convective 
weather  and  little  or  no  awareness  of  the  Product  itself. 

2.  How  did  the  NCWF  compare  to  other  convective  weather  products? 

Many  users  (7)  compared  the  Product  to  the  current  Weather  Information  Dissemination  Service 
(WIDS)  display.  Although  Product  update  times  were  viewed  favorably,  the  Product’s  inability  to 
zoom  and  provide  animation  was  noted.  At  least  six  respondents  compared  the  NCWF  Product 
favorably  to  other  convective  weather  sources.  One  noted  that  it  provided  a  better  definition  of 
convective  areas  than  Convective  SIGMETs. 

3.  Did  the  NCWF  Product  add  value? 

Most  dispatchers  (13)  indicated  the  Product  did  add  value  in  performance  of  duties  affected  by 
convective  weather  adding  the  graphical  projection  was  useful  information.  Three  indicated  the 
Product  added  no  value  due  to  insufficient  forecast  duration  (a  1 -hour  forecast  was  considered 
inadequate  for  planning  purposes)  and  lack  of  confidence  in  Product  accuracy. 

4.  Did  Product  use  save  time? 

Several  dispatchers  (8)  stated  that  NCWF  Product  use  did  not  save  them  time.  However,  they 
felt  that  timesaving  was  not  as  important  as  information  accuracy. 

5.  What  NCWF  Product  components  were  most  useful? 

Components  found  most  useful  included  the  1-hour  forecast  capability  and  echo  speed 
information. 

6.  Describe  typical  situations,  if  any,  where  the  NCWF  Product  and/or  its  components 
performed  well. 

At  least  eight  dispatchers  could  not  recall  instances  where  the  NCWF  Product  did  or  did  not 
perform  well.  One  positive  Product  scenario  involved  a  pilot  holding  over  a  fix  due  to  severe 
convective  weather  activity.  By  using  the  NCWF,  the  dispatcher  reported  seeing  a  “window  of 
opportunity”  and  could  see  the  storm  was  moving  off.  The  aircraft  successfully  flew  through 
without  diverting  and  refueling.  Another  scenario,  received  over  the  Product  comment  daily  log, 
reported  that  “the  algorithm  nailed  the  progress  (thunderstorms)  perfectly  and  its  guidance 
allowed  me  to  make  the  correct  decision  with  regard  to  the  operational  control  of  my  flight.” 
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7.  Describe  typical  situations,  if  any,  where  the  NCWF  Product  and/or  its  components 
performed  poorly. 

Again,  most  interviewed  could  not  recall  specific  Product  performance  scenarios.  One 
respondent  cited  an  instance  where  a  storm  was  forecast  to  move  south,  but  actually  moved 
east.  Another  reported  a  line  moving  south  to  Chattanooga  was  not  detected  at  all.  It  was  also 
noted  the  Product  performed  well  in  clearly  defined  frontal  situations  but  not  with  air  mass 
thunderstorms. 

8.  Would  Product  animation  add  more  value?  Would  a  longer  forecast  time  be  preferable? 

All  dispatchers  agreed  that  animation  and  longer  forecast  durations  would  be  beneficial.  All 
were  aware  that  given  a  longer  forecast  duration,  degradation  in  accuracy  may  be  an  issue. 
Although  difficult  to  measure,  many  agreed  that  degradation  over  50  percent  would  not  be 
acceptable. 

9.  Additional  Suggestions  for  Improvement 

a.  Provide  individual  zoom  capability  at  a  user  selected  area. 

b.  Include  animation  for  trending. 

c.  Include  annotated  speed  of  the  storm  tops. 

d.  Integrate  the  forecast  component  with  the  ASD  display  for  a  bigger  picture,  thereby 
negating  the  need  for  mentally  overlaying  geographic  identifiers,  radar  locations,  and  aircraft 
positions. 

Delta  Interview  comment  summaries  may  be  referenced  in  appendix  C. 

5.2.2  Northwest  Airlines  Interview  Results. 

Seven  users,  including  dispatchers  and  dispatch  coordinators  were  interviewed.  Interviews  took 
place  in  the  dispatch  operations  area.  A  brief  summary  is  discussed  after  each  interview 
question  in  italics  below. 

1.  Did  you  use  the  NCWF  Product? 

Most  dispatchers  (5)  reported  using  the  Product  minimally,  or  looking  at  it  nonoperationally. 
Generally,  this  was  attributed  to  a  lack  of  significant  convective  weather  and  lack  of  confidence 
in  the  Product’s  accuracy.  Two  dispatch  coordinators,  however,  reported  extensive  operational 
NCWF  Product  use  for  tasks  involving  Collaborative  Decision  Making  (CDM),  fuel  planning,  and 
rerouting. 

2.  How  did  the  NCWF  Product  compare  to  other  convective  weather  products? 

Comparisons  to  other  weather  Products  were  divided.  NCWF  was  compared  mostly  to  the 
ASD,  which  contains  location  identifiers,  aircraft  positions,  and  current  animated  radar 
information.  Suggestions  for  combining  the  two  were  often  noted.  One  dispatcher  noted  that  he 
had  been  mentally  drawing  the  forecast  lines  for  years  and  found  the  NCWF  Product  very 
effective.  Conversely,  another  dispatcher  stated  the  Product  was  worthless  and  compares 
poorly  to  current  radars,  Airline  Dispatch  Federation  (ADF)  website  information,  and  composite 
radar  and  satellite  displays. 
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3.  Did  the  NCWF  Product  add  value? 

Most  dispatchers  (5)  agreed  the  NCWF  Product  had  value  -  or  could  potentially  add  value  to 
current  dispatch  operations.  One  dispatcher  indicated  the  Product  would  not  add  value  and 
another  could  not  comment  on  this  question. 

4.  Did  Product  use  save  time? 

Three  dispatchers  reported  the  Product  has  the  potential  to  save  time  although  they  did  not 
experience  any  timesaving.  One  dispatcher  noted  it  would  not  save  time,  whereas  the 
remaining  dispatchers  (3)  felt  they  could  not  respond  to  this  question. 

5.  What  NCWF  Product  components  were  most  useful? 

Overall,  the  forecast  component  was  found  most  useful. 

6.  Describe  typical  situations,  if  any,  where  the  NCWF  Product  and/or  its  components 
performed  well. 

Two  dispatchers  reported  specific  instances  where  the  Product  performed  well.  In  one  instance 
thunderstorms  had  developed  along  a  route  from  Minneapolis  to  Sioux  Falls  and  was  tracking 
northward.  By  using  the  information  from  the  NCWF  forecast  polygons,  the  dispatcher  could 
effect  shortcuts  by  routing  flights  south  of  Sioux  Falls.  The  forecast  was  accurate,  and  aircraft 
were  able  to  get  through.  In  another  scenario,  a  dispatch  coordinator  reported  thunderstorms 
were  impacting  the  Western  routes  from  Minneapolis.  By  using  the  NCWF,  he  was  able  to 
anticipate  ATC  actions  and  plan  reroutes. 

7.  Describe  typical  situations,  if  any,  where  the  NCWF  Product  and/or  its  components 
performed  poorly. 

One  dispatcher  commented  that  in  some  instances,  cell  development  was  not  displayed  even 
though  significant  convective  activity  was  present.  Two  dispatchers  indicated  that  lack  of 
growth  and  decay  capability  was  a  serious  limitation  to  the  Product. 

8.  Would  Product  animation  add  more  value? 

All  dispatchers  interviewed  reported  that  animation  would  be  beneficial  in  order  to  see  trending. 

9.  Would  a  longer  forecast  time  be  preferable? 

All  dispatchers  interviewed  agreed  that  a  longer  forecast  time  would  be  preferable  with  a 
minimum  duration  of  2  hours.  Degradation  in  accuracy  would  be  acceptable  if  the  degree  of 
accuracy  were  known. 
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10.  Additional  Suggestions  for  Improvement 


a.  Prediction  of  thunderstorm  movement  with  a  high  degree  of  probability. 

b.  Longer  forecasts  -  up  to  3  hours. 

c.  Addition  of  a  growth  and  decay  component. 

d.  Provide  user  selected  zoom  capability. 

e.  Overlay  aviation  identifiers  such  as,  navigational  aids  (NAVAIDS),  aircraft  position  and 
jetways  as  well  as  geographical  locators  as  displayed  on  the  ASD.  Allow  user  to  toggle 
overlays  on  and  off. 

f.  Overlay  the  Product  forecast  component  on  the  existing  ASD. 

g.  Provide  consistent  symbology  with  other  weather  systems. 

h.  Display  instability  and  weather  fronts. 

i.  Ensure  Product  annotations  (speed  and  tops)  do  not  obscure  weather  depictions. 
Northwest  interview  summaries  are  located  in  appendix  C. 

5.2.3  Atlantic  Coast  Airlines  Interview  Results. 

Six  dispatchers  were  interviewed  at  the  Atlantic  Coast  Airlines  dispatch  operations  center. 
Comments  are  summarized  below  under  each  corresponding  question.  See  appendix  C  for  all 
interview  question  responses. 

1.  Did  you  use  the  NCWF  Product? 

All  six  dispatchers  reported  using  the  NCWF  Product  operationally.  It  was  noted,  however,  that 
Product  inaccessibility  was  an  issue.  One  dispatcher  reported  using  the  NCWF  for  short  haul 
flights,  especially  in  the  Northeast  corridor,  where  it  helped  to  see  where  convective  weather 
would  be  impacting  in  an  hour. 

2.  How  did  the  NCWF  compare  to  other  convective  weather  products? 

Most  dispatchers  viewed  the  forecast  component  of  the  NCWF  Product  as  another  convective 
weather  information  source  used  in  conjunction  with  available  weather  sources  such  as  terminal 
aerodrome  forecasts  (TAF),  radar,  and  National  Weather  Service  (NWS)  briefings.  Some 
reported  the  forecast  helped  with  the  guesswork  and  allowed  them  to  see  a  bigger  weather 
picture. 

3.  Did  the  NCWF  Product  add  value? 

All  dispatchers  interviewed  agreed  the  NCWF  Product  had  value  -  or  could  potentially  add 
value  to  current  dispatch  operations.  Potential  value  areas  included  safety,  planning,  and 
increased  revenues. 

4.  Did  Product  use  save  time? 

Two  dispatchers  reported  the  Product  had  saved  time  operationally.  One  stated  it  had  the 
potential  to  save  time  after  extended  use.  Another  noted  that  when  NCWF  Product  confidence 
increases,  so  will  timesavings.  Two  dispatchers  did  not  comment  on  this  question. 
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5.  What  NCWF  Product  components  were  most  useful? 

Overall,  the  forecast  component  was  found  most  useful. 

6.  Describe  typical  situations,  if  any,  where  the  NCWF  Product  and/or  its  components 

performed  well. 

One  dispatcher  noted  the  NCWF  seemed  to  work  best  when  there  was  a  solid  line  of 
thunderstorms.  In  one  scenario,  storms  from  Canada  to  North  Carolina  were  cutting  off  the 
Northern  routes.  The  NCWF  displayed  gaps  in  the  lines  large  enough  for  aircraft  to  fly  through. 
This  information  was  also  useful  for  coordination  activities  with  the  Air  Traffic  Control  System 
Command  Center  (ATCSCC).  Planning  was  more  proactive  (5  minutes  versus  1  hour  ahead). 
Another  dispatcher  reported  thunderstorms  developing  Northwest  from  Cleveland  through  the 
Pittsburgh  area  and  moving  south.  Although  the  radar  animation  helped  gauge  storm 
movement,  the  NCWF  was  more  specific  in  identifying  locations  and  helped  in  making  better 
routing  decisions. 

7.  Describe  typical  situations,  if  any,  where  the  NCWF  Product  and/or  its  components 
performed  poorly. 

Although  specific  instances  were  not  recalled,  some  noted  the  algorithm  did  not  pick  up  isolated 
cells  and  missed  smaller  storms  identified  by  the  ASD. 

8.  Would  NCWF  Product  animation  add  more  value ? 

All  dispatchers  interviewed  reported  that  animation  would  be  beneficial  in  order  to  see  trending. 

9.  Would  a  longer  forecast  time  be  preferable? 

All  dispatchers  interviewed  agreed  that  a  longer  forecast  time  would  be  preferable  with  a 
minimum  duration  of  2  hours.  Degradation  in  accuracy  of  30  —  40  percent  would  be  acceptable 
for  a  4-  to  6-hour  forecast. 

10.  Additional  Suggestions  for  Improvement 

The  following  suggestions  for  improvement  were  mentioned  during  the  course  of  the  interview 
and  noted  below: 

a.  Overlay  the  NCWF  forecast  component  on  the  ASD, 

b.  Provide  annotated  cloud  heights  information, 

c.  Provide  1-  to  2-hour  animations  for  trending, 

d.  Allow  user  selected  zoom  capability, 

e.  Incorporate  lightning  data  as  a  separate  component. 


6.  CONCLUSIONS. 

With  few  exceptions,  overall  feedback  from  participating  dispatchers  was  positive  and  overall 
ratings  between  airlines  were  consistent.  A  summary  of  results  from  the  1 998  National 
Convective  Weather  Forecast  (NCWF)  Summer  Demonstration,  corresponding  to  each  of  the 
stated  Demonstration  objectives  (in  italics),  are  discussed  below. 
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1.  Does  the  NCWF  Product  add  value  to  identifying  and  forecasting  areas  of  convective 
weather  activity? 

When  asked  directly  about  Product  value,  most  dispatchers  agreed  the  NCWF  Product  had  the 
potential  to  add  value  in  identifying  and  forecasting  areas  of  convective  weather  activity.  The 
tendency  to  state  “potential  value”  as  opposed  to  “actual  value”  was  based  on  infrequency  of 
use  and  confidence  in  the  accuracy  of  the  detections  and  forecast.  Until  confidence  is  gained, 
dispatchers  are  generally  reluctant  to  rely  on  information  from  an  experimental  source. 

Additionally,  most  reports  on  positive  NCWF  Product  performance  indicated  the  Product 
enabled  the  dispatcher,  on  some  occasions,  to  see  breaks  within  a  line  of  thunderstorms, 
enabling  them  to  fly  planes  through  instead  of  rerouting.  This  is  also  indicated  via  the  high 
usefulness  ratings  in  tasks  involving  (1)  advising  pilots  enroute  of  severe  convective  weather, 
and  (2)  flight  monitoring. 

2.  Does  the  NCWF  Product  benefit  tasks  and  decisions  affected  by  convective  weather 
activity? 

Based  on  questionnaire  data  relating  Product  usefulness  to  job  task  performance  overall,  the 
Product  received  positive  to  very  positive  ratings.  Product  usefulness  was  rated  highest  (of 
considerable  use)  in  5  of  the  1 1  task  areas: 

a.  Advising  pilots  enroute  of  severe  convective  weather, 

b.  Avoidance  of  severe  convective  weather, 

c.  Routing  flights, 

d.  Planning  the  flight  route, 

e.  Monitoring  flight  conditions. 

Conversely,  NCWF  Product  usefulness  was  rated  borderline  overall  in  three  task  areas: 

a.  Selecting  alternate  airports, 

b.  Anticipating  ground  delays, 

c.  Anticipating  air  traffic  control  (ATC)  actions. 

The  borderline  rating  indicates  the  NCWF  Product  had  neither  a  positive  nor  negative  effect  in 
the  performance  of  these  tasks. 

The  remaining  four  task  areas  received  positive  usefulness  ratings  and  included: 

a.  Preparing  flight  plans, 

b.  Planning  fuel  quantity, 

c.  Selecting  altitude, 

d.  Anticipating  ATC  actions. 

Again,  results  indicate  the  NCWF  Product  is  useful  in  performing  these  tasks. 


27 


3.  How  well  does  the  Product  perform?  Under  what  situations  does  the  Product  perform  well  or 
not  perform  well? 

Most  dispatchers  reported  the  Product  performed  well.  This  was  particularly  true  during  well- 
defined  frontal  situations.  Positive  Product  performance  was  also  indicated  by  the  positive 
confidence  ratings  from  questionnaire  data  on  Product  components  such  as  initial  detection  and 
forecast  contours. 

Reports  of  poor  Product  performance  centered  on  the  NCWF’s  occasional  inability  to  detect  and 
track  some  storms  such  as  air  mass  thunderstorms  and  incorrectly  display  storm  movement. 
Absence  of  a  growth  and  decay  component  and  longer  forecast  durations  were  also  considered 
Product  limitations. 

4.  How  can  the  Product  be  enhanced?  What  improvements  or  changes  would  this 
necessitate? 

Although  many  suggestions  for  improvements  or  changes  were  suggested,  the  following 
represent  items  reported  most  often: 

a.  Animation  -  The  capability  to  loop  images  from  1  to  2  hours  to  see  weather  trends  was 
expressed  by  almost  all  dispatchers  in  both  interview  and  questionnaire  responses. 

b.  User  selected  zoom  function  —  Most  dispatchers  indicated  the  need  for  better  zoom 
capability  and  considered  available  Product  views  too  limited.  This  was  reported  in  both  open- 
ended  questionnaire  responses  and  during  interview  sessions. 

c.  Longer  forecast  duration  -  Most  dispatchers  considered  the  1-hour  forecast  insufficient. 
Flight  plans  must  be  issued  to  the  pilot  at  least  1  hour  or  more  prior  to  takeoff.  A  1-hour 
forecast,  therefore,  is  inadequate  for  flight  planning  considerations.  However,  based  on 
questionnaire  responses,  the  Product  did  receive  overall  positive  usefulness  ratings  for 
preparing  flight  plans  and  planning  flight  routes. 

d.  Growth  and  decay  component  -  Since  offset  and  onset  of  weather  is  of  particular 
interest  to  dispatchers  for  flight  planning,  this  would  augment  and  enhance  overall  perceived 
performance  of  the  NCWF.  Some  dispatchers  noted  that  offset  of  weather  was  of  particular 
value  for  dispatcher  planning  purposes.  Knowing  when  and  where  a  storm  will  dissipate  was 
viewed  as  more  critical  than  knowing  where  it  will  develop. 

e.  Overlays  -  The  NCWF  was  often  compared  to  the  aircraft  situation  display  (ASD),  which 
is  used  extensively  for  flight  monitoring  and  identifying  current  weather.  The  ASD  overlays 
consist  of  geographical  locators  and  identifiers  such  as  airports,  very  high  frequency  omni-range 
radars  (VOR),  arrival/departure  gates,  and  airplane  locations.  These  types  of  overlays,  if  user 
selected,  would  obviate  the  need  for  dispatchers  to  mentally  overlay  identifiers. 

5.  Is  a  1-hour  forecast  Product  adequate  for  airline  users’  needs? 

As  noted  above,  the  1-  hour  forecast  is  not  adequate  for  dispatcher  planning  purposes. 
Degradation  in  forecast  accuracy  was  acceptable  as  long  as  it  did  not  exceed  50  percent  and 
accuracy  constraints  were  understood;  i.e.,  the  user  is  aware  of  lower  accuracy  thresholds  as 
the  forecast  is  extended.  A  3-6  hour  forecast  was  preferred  overall. 
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6.  Are  Product  interfaces  and  characteristics  operationally  acceptable? 

Based  on  questionnaire  rating  data,  overall  Product  interfaces  such  as:  readability  and  utility  of 
color  scales;  detection  fields;  forecast  contours;  overlays;  views;  navigation;  and  update  rates 
were  rated  positively  (acceptable  to  largely  acceptable).  However,  some  comments  from  open 
ended  questionnaire  questions  and  interview  questions  indicated  that  navigation  between  views 
could  be  simplified,  additional  overlays  and  information  are  needed,  and  care  should  be  taken 
not  to  obscure  weather  data  with  echo  speed  and  height  information.  Update  rates  were  also 
criticized  as  being  too  long.  Although  Product  views  were  rated  positively,  many  reported  the 
Air  Route  Traffic  Control  Center  (ARTCC)  and  Regional  views  were  inadequate  for  monitoring 
and  planning  flights,  indicating  the  need  for  a  user  selected  zoom  capability. 


7.  RECOMMENDATIONS. 

a.  Ensure  training  is  consistent  and  across  the  board  by  employing  “train  the  trainer” 
methodologies  to  reach  a  larger  population.  Although  use  of  the  National  Convective  Weather 
Forecast  (NCWF)  Product  was  intuitive,  knowledge  of  its  capabilities  and  limitations  may  have 
increased  Product  use  overall. 

b.  Emphasis  should  be  placed  on  future  Product  improvements  based  on  suggestions 
reported  and  discussed  in  section  6.  These  include: 

1 .  Animation, 

2.  User  selected  zoom  function, 

3.  Longer  forecast  durations  (3-6  hours), 

4.  Growth  and  decay  component, 

5.  Additional  overlays  and  geographical  identifiers. 

Additions  of  any  or  all  of  these  components  may  enhance  the  Product’s  overall  usefulness  in 
meeting  dispatch  requirements  for  an  automated,  graphical,  convective,  forecast  and  detection 
tool. 

c.  Additional  research  is  recommended  to  increase  performance  under  certain  conditions; 
particularly  when  storms  are  not  well  defined  or  when  air  mass  thunderstorms  are  present. 

d.  Given  the  total  number  of  dispatchers  (approximately  300)  at  the  three  participating 
airlines,  overall  response  was  poor.  Methods  for  increasing  participation  of  users  should  be 
researched. 
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8.  ACRONYMS. 

ACA 

_ 

Atlantic  Coast  Airlines 

ACSF 

- 

Automated  Convective  SIGMET  Forecast 

ADF 

- 

Aviation  Dispatch  Federation 

AP 

- 

Anomalous  Propagation 

ASD 

- 

Aircraft  Situation  Display 

ARTCC 

- 

Air  Route  T raffic  Control  Center 

ATC 

- 

Air  Traffic  Control 

ATCSCC 

- 

Air  Traffic  Control  System  Command  Center 

AWC 

- 

Aviation  Weather  Center 

AWR 

- 

Aviation  Weather  Research 

CDM 

- 

Collaborative  Decision  Making 

MDV 

- 

Meteorological  Data  Volume  (NCAR  internal) 

MIT/LL 

- 

Massachusetts  Institute  of  Technology/Lincoln  Labs 

NAVAIDS 

- 

navigational  aids 

NCAR 

- 

National  Center  for  Atmospheric  Research 

NCWF 

- 

National  Convective  Weather  Forecast 

NLDN 

- 

National  Lightning  Detection  Network 

nmi 

- 

nautical  miles 

NSSL 

- 

National  Severe  Storms  Laboratory 

NWS 

- 

National  Weather  Service 

SPDB 

- 

SYMPROD  database  (NCAR  internal) 

TAF 

- 

Terminal  Aerodrome  Forecast 

WIDS 

- 

Weather  Information  Dissemination  Service 

VIL 

- 

Vertically  Integrated  Liquid 

VOR 

- 

Very  High  Frequency  Omni-Range 
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APPENDIX  A 


DISPATCHER  QUESTIONNAIRE 


1998  NATIONAL  CONVECTIVE  WEATHER 
FORECAST  DEMONSTRATION 

DISPATCHER 

QUESTIONNAIRE 


Prepared  by: 

Communication/Navigation/Surveillance 
Engineering  and  Test  Division,  Weather  Branch,  ACT-320 
William  J.  Hughes  Technical  Center 
Federal  Aviation  Administration 
Atlantic  City  International  Airport 
Atlantic  City,  NJ  08405 
and 

National  Center  for  Atmospheric  Research 
Research  Applications  Program 
3450  Mitchell  Lane 
Boulder,  CO  80301 


Please  provide  the  following  information: 

Airline: _ Title:. 

Please  check  or  indicate  the  dispatch  area  you  cover 

Long  Haul _  Short  Haul _  Other. 


:  if!  Instructions  IBB 


The  purpose  of  this  questionnaire  is  to  obtain  feedback  from  airline  dispatchers  regarding  the 
National  Convective  Weather  Forecast  (NCWF)  Product.  The  demonstration  of  the  NCWF 
Product  is  being  conducted  by  the  National  Center  for  Atmospheric  Research  (NCAR)  with  the 
assistance  of  the  FAA’s  William  J.  Hughes  Technical  Center. 

Feedback  from  dispatchers  is  a  very  important  component  of  the  demonstration,  and  your 
responses  to  this  questionnaire  will  provide  important  information  for  use  in  future  iterations  of  the 
NCWF  Product.  Please  respond  to  all  questions  as  honestly  and  thoroughly  as  possible. 

All  responses  will  remain  ANONYMOUS  and  CONFIDENTIAL.  A  report  will  be  written  on  the 
results  of  this  questionnaire,  summarizing  respondent’s  comments;  however,  no  one  will  be 
identified  or  associated  with  any  specific  comment.  Please  return  the  questionnaire  to  the 
FAA’s  Technical  Center  Evaluator. 


A-2 


The  five-point  scale  below  should  be  used  to  rate  the  National  Convective  Weather  Forecast  Product. 

The  following  definitions  are  provided  for  use  when  assessing  the  Product.  Please  refer  to  these 

definitions  when  responding.  Feel  free  to  tear  out  this  page  and  keep  it  as  a  reference  when  answering 

questions.  Also,  for  your  reference,  a  color  print  of  the  NCWF  Product  display  has  been  provided  to  aid 

in  relating  specific  questions  to  the  components  of  the  display. 

(1)  Largely  Acceptable  -  This  response  indicates  the  Product/component  being  assessed  consistently 
enhances  your  ability  to  meet  the  requirements  of  your  job;  likely  to  lead  to  enhanced  job 
performance. 

(2)  Acceptable  -  This  response  indicates  the  Product/component  being  assessed  frequently  enhances 
your  ability  to  meet  the  requirements  of  your  job;  may  lead  to  enhanced  job  performance. 

(3)  Borderline  -  This  response  indicates  that,  although  the  Product/component  being  assessed  is 
adequate,  minor  improvements  would  make  it  more  helpful  in  identifying  and  forecasting  convective 
weather  and  does  not  degrade  or  improve  job  performance. 

(4)  Unacceptable  -  This  response  indicates  the  Product/component  being  assessed  frequently  impedes 
your  ability  to  meet  the  requirements  of  your  job:  may  lead  to  degradation  of  job  performance. 

(5)  Largely  Unacceptable  -  This  response  indicates  the  Product/component  being  assessed 
consistently  impedes  your  ability  to  meet  the  requirements  of  your  job;  likely  to  lead  to  degradation  of 
job  performance. 

NA  -  you  have  never  used  the  Product/component  in  question. 


You  will  be  asked  to  rate  the  Product  and  its  components  according  to  their  Utility,  Reliability, 
Readability,  and  Confidence.  The  following  definitions  should  be  considered  when  answering  questions. 


(1)  Utility  -  The  Product/  component  is  useful,  meets  job  requirements  and  responsibilities,  and  aids  in 

identifying  and  forecasting  areas  of  convective  weather; 

(2)  Reliability  -  The  Product/components  are  consistently  working  without  problems.  Data  is  provided 
consistently; 

(3)  Readability  -  This  refers  to  how  readable  or  legible  the  Product  or  feature  is  (i.e.  display  clutter,  font 
size,  color-coding). 

(4)  Confidence  -  This  refers  to  the  degree  of  user  confidence  based  on  the  Product’s  ability  to  detect 
and  forecast  convective  weather  areas. 


Instructions:  Please  rate  the  National  Convective  Weather  Forecast  Product  and  its 
components  by  circling  the  appropriate  number. 


1 .  Initial  Detection  Fields 

a.  Utility  1 

b.  Reliability  1 

c.  Readability  1 

d.  Confidence  1 


2  3  4 
2  3  4 
2  3  4 
2  3  4 


5  NA 

5  NA 

5  NA 

5  NA 


2.  Convective  Intensity  Color  Scale 

a.  Utility 

b.  Reliability 

c.  Readability 

3.  Detection  Field  Colors 

a.  Utility 

b.  Readability 

4.  Forecast  Contours 

a.  Utility 

b.  Reliability 

c.  Readability 

d.  Confidence 


1  2  3 
1  2  3 
1  2  3 

1  2  3 
1  2  3 

1  2  3 
1  2  3 
1  2  3 
1  2  3 


4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 


5.  Forecast  Direction  Lines 

a.  Utility 

b.  Reliability 

c.  Readability 

d.  Confidence 

6.  50  nm  Range  Rings 

a.  Utility 

b.  Readability 

7.  Storm  Height  Information 

a.  Utility 

b.  Reliability 

c.  Readability 

d.  Confidence 


1  2  3 
1  2  3 
1  2  3 
1  2  3 

1  2  3 
1  2  3 

1  2  3 
1  2  3 
1  2  3 
1  2  3 


4  5  NA 

4  5  l$A 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 
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National  Convective 

Weather  Forecast  Product 

8.  Echo  Speed  Values 

So  A 

Bo 

cceptable 

rderiine 

§:|§ 

; 

Unaccepts 

ible  :  Unacceptable 

iff® 

a.  Utility 

1 

2 

3 

4 

5 

NA 

b.  Reliability 

1 

2 

3 

4 

5 

NA 

c.  Readability 

1 

2 

3 

4 

5 

NA 

d.  Confidence 

1 

2 

3 

4 

5 

NA 

9.  ARTCC  Boundary  Overlays 

a.  Utility 

1 

2 

3 

4 

5 

NA 

b.  Reliability 

1 

2 

3 

4 

5 

NA 

10.  ARTCC  Map 

a.  Utility 

1 

2 

3 

4 

5 

NA 

b.  Reliability 

1 

2 

3 

4 

5 

NA 

1 1 .  Views  (National,  Eastern,  et  al) 

a.  Utility 

1 

2 

3 

4 

5 

NA 

b.  Reliability 

1 

2 

3 

4 

5 

NA 

12.  Regional  Views 

a.  Utility 

1 

2 

3 

4 

5 

NA 

b.  Reliability 

1 

2 

3 

4 

5 

NA 

13.  Navigation  Between 

Views 

1 

2 

3 

4 

5 

NA 

14.  Product  Update  Rate 

1 

2 

3 

4 

5 

NA 

15.  List  suggestions  for  improving  any  of  the  above  Products,  functions  or  displayed  items.  Please  be 
sure  to  include  the  item  name  in  the  suggestion(s). 


16.  List  other  attributes  or  components  that  you  think  should  be  added  to  the  National  Convective 
Weather  Forecast  Product. 
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mt 


HUMP' for  Jo*>  fas 


Instructions:  The  five  point  scale  below  should  be  used  to  rank  the  usefulness  of  the  National 
Convective  Weather  Forecast  Product  as  related  to  convective  weather,  on  each  of  the  tasks  listed. 
Please  refer  to  these  definitions  when  responding. 


IS 


1  -  Of  Considerable  Use.  This  response  indicates  the  NCWF  Product  had  a  significant  positive  effect 
on  decisions  and  activities  related  to  the  performance  of  this  task. 


2  -  Somewhat  Useful.  This  response  indicates  the  NCWF  Product  had  a  positive  effect  on  decisions 
and  activities  related  to  the  performance  of  the  task. 

3  -  Borderline.  This  response  indicates  the  NCWF  Product  had  little  to  no  effect  on  decisions  and 
activities  related  to  the  performance  of  the  task. 

4  -  Of  Little  Use.  This  response  indicates  the  NCWF  had  a  negative  effect  on  decisions  and  activities 
related  to  the  performance  of  the  task. 

5.  -Not  at  all  Useful.  This  response  indicates  the  NCWF  Product  had  a  significant  negative  effect  on 
decisions  and  activities  related  to  the  performance  of  the  task. 


NA  —  You  have  never  used  the  Product  in  performing  this  task. 


19.  Planning  flight  route.  1 

20.  Avoidance  of  severe  1 

convective  weather. 

21 .  Selecting  an  alternate  airport.  1 

22.  Selecting  altitude.  1 


2 

2 

2 

2 


3 

3 

3 

3 


4  5  NA 

4  5  NA 

4  5  NA 

4  5  NA 


23.  Monitoring  flight  conditions.  1 

24.  Re-routing  flights.  1 

25.  Advising  pilots  enroute  of  1 

severe  convective  weather. 

26.  Anticipating  ground  delays.  1 


2 

2 

2 


3  4  5  NA 

3  4  5  NA 

3  4  5  NA 


2 


3 


4  5  NA 


27.  Anticipating  ATC  actions  (i.e.  1 

rerouting,  closing  runways, 

changing  arrival  gates) 

28.  Other _ _  1 


2 


3 


4  5  NA 


2 


3 


4  5  NA 
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Thank  you  for  your  time  and  cooperation.  Please  mail  completed  questionnaires  to: 


Cynthia  Fidalgo 
Raytheon  Systems  Co. 

500  Scarborough  Drive  #304 
Egg  Harbor  Township,  NJ  08405 
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APPENDIX  B 


QUESTIONNAIRE  COMMENT  RESULTS 


Northwest,  Delta,  and  Atlantic  Coast  Airlines  (ACA)  Questionnaire  Responses  -  #  15  and  16 

4 

15.  List  suggestions  for  improving  any  of  the  above  Products,  functions  or  displayed  items. 


Response 

State  borders  are  nice  but  in  trying  to  provide  location  information  to  pilots  we  speak  in  terms  of  jet  routes,  or  distance  from 
airport  or  navaid.  Would  like  to  be  able  to  zoom  to  a  specific  location  vs.  a  pre-set  area  or  region. 

Detection  should  include  direction  and  speed  of  thunderstorm  area.  Echo  tops  should  also  be  included. 

Click  and  zoom  function,  buttons  are  cumbersome  and  increase  update  time.  Nav  data  overlays  need  to  be  added  (cities, 
VORs,  jet  airways). 

ARTCC  boundaries  and  maps  need  a  more  applicable  basemap  including  navaids,  fixes,  airways,  sectors  as  well  as  ARTCC 
boundaries  and  airports.  Also  the  ability  to  select  the  above  elements  on/off  would  help. 

Update  rate  needs  to  be  improved  greatly. 

Timely  to  use  in  our  present  version  that  was  available  to  me. 

One  hour  forecasts  are  relatively  of  little  value.  Average  NWA  segment  is  1:15.  Average  release  is  sent  2+  hours  in 
advance.  We  need  3  hour  forecast! 

The  viewer  interface  was  difficult  to  use  and  made  it  hard  to  navigate  to  the  information  needed.  Need  to  be  able  to  zoom 
into  selected  areas  for  viewing. 

Navigation  between  views  could  be  simplified  or  defined  better. 

1 

1 

Would  like  to  lightning  detection  separate  so  you  can  see  them  or  overlay  with  +  or  -  marks.  14.  Need  to  be  able  to  project 
movement  of  TRW  -2/3/4  hrs. 

Display  very  useful  8/9/98.  Color  change  on  display  for  forecast  contours  and  speed/tops  data  made  speed/tops  unreadable. 

Blank 

Be  able  to  zoom  and  loop  images.  Also,  1  hr.  is  not  acceptable  for  a  forecasting  tool.  We  need  4-6  hrs. 

Font  size  could  be  larger.  Direction  arrows  longer. 

1  would  like  to  further  enlarge  areas  of  convective  activity.  There  should  be  additional  airport  zooms  CX  NYC  area,  ATC,  DCA 
area. 

We  should  be  able  to  zoom  in  and  out. 

Extend  forecast  range  on  separate  view  to  2  or  3  hours. 

When  it  updates,  1  do  not  like  the  way  it  updates  the  US  map  and  displays  it,  then  it  goes  to  whatever  you  have  selected  and 
updates  that  map. 

Loop  current  returns  and  project  out  4  to  6  hrs.  To  help  with  long  haul  planning.  Show  areas  diminishing  as  thev  do. 

The  display  of  storm  height  information  and  echo  speed  values  does  not  conform  to  convential  displays  normally  used  with 
this  information.  This  is  initially  confusing.  Perhaps  adding  the  identifiers  "MT  xxx"  and  "xxkts"  would  make  the  nature  of 
these  val 

Need  looping  to  re-enforce  the  confidence  level  in  the  forecast,  ie  time  lapse. 
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16.  List  other  attributes  or  components  that  you  think  should  be  added  to  the  NCWF  Product 
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APPENDIX  C 


STRUCTURED  INTERVIEW  QUESTIONS 


Interview  Questions  for  the  National  Convective  Weather  Forecast  Product  Demonstration 

1 .  Did  you  use  the  Product  for  identification  and  forecasting  of  convective  weather  areas?  Please 
explain  why  or  why  not. 

2.  How  did  the  NCWF  information  compare  to  other  convective  weather  information  sources? 

3.  Did  the  Product  add  value  in  performance  of  your  duties  effected  by  convective  weather? 

4.  Did  Product  use  save  time? 

5.  What  Product  components,  if  any,  did  you  find  most  useful?  Why? 

6.  Describe  typical  situations,  if  any,  where  the  Product  and/or  its  components  performed  well. 

7.  Describe  typical  situations,  if  any,  where  the  Product  and/or  its  components  performed  poorly. 

8.  Would  Product  animation  add  more  value? 

9.  Would  a  longer  forecast  time  be  preferable? 

-  For  your  needs,  what  would  be  the  ideal  forecast  time? 

10.  What  degree  of  accuracy  is  required  for  a  2  or  more  hour  forecast  Product?  i.e.:  would  a  2  hour 
forecast  have  to  be  as  accurate  as  a  1  hour  forecast? 

-  What  amount  of  degradation  in  forecast  accuracy  would  be  acceptable  for  the  following  forecast 
times? 


hour  forecast  - 10% 

20% _ 

_  30%  _ 

_  40% _ 

_  50%. 

hour  forecast -10% 

20% _ 

_  30%  _ 

_  40% _ 

_  50%. 

hour  forecast -10% 

20% _ 

__  30%  _ 

_  40% _ 

_  50%. 
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APPENDIX  D 


STRUCTURED  INTERVIEW  SUMMARIES 


INTERVIEW  RESPONSES  -  NORTHWEST,  DELTA  AND  ATLANTIC  COAST  AIRLINES  (ACA) 
Did  you  use  the  product  for  identification  and  forecasting  of  convective  weather  areas?  Please  explain  why  or  why  not. 


Response 

Have  not  looked  at  it  yet. 

Not  used  -  knew  it  was  here,  but  did  not  use.  Wanted  to  wait  until  it  was  fine  tuned.  Would  be  nice  to  filter  out  level  1  and  2. 

Used  it.  -  but  data  was  4  hours  old.  Did  not  use  much  after  that.  A  few  times. 

Uses  it  -  Almost  always  up. 

Use  when  convective  wx  present  -  use  quite  often. 

Yes.  3  or  4  days  used  forecast  contours. 

Yes  -  Occasionally  used  once  -  mostly  looked  at. 

Use  everyday  -  like  looking  ahead. 

Used  occasionally  -  sometimes  looks  at. 

Used  often  -  generally  up.  | 

Used  for  1  Week. 

Used  5x  -  Used  to  see  projections  for  TSs  - 1  hr  ahead.  1 

Used  quite  a  bit  -  Make  font  bigger.  j 

Uses  all  the  time  -  helps  with  holding,  anticipating  holding  -  wx  at  fixes  anticipated  ATS  action.  Would  like  to  see:  Customize 
picture,  zoom  on  cities,  include  zoom  box.  Can  see  breaks  in  relation  to  position  for  pilot.  Better  than  pilot’s  radar. 

Uses  product  all  the  time.  Problem:  Need  zoom  -  area  1  am  interested  in.  Box  to  zoom  in  on.  Can  customize  zoom.  Can’t  see 
picture  of  all  Ts  activity.  Need  to  manipulate  display  too  much.  Need  to  use  slider.  Would  like  full  view  of  display.  Savef 

Used  it  -  Almost  everyday  when  had  wx  is  present. 

Used  but  not  as  much  as  the  ASD  display.  Did  use  during  diverse  weather  to  see  where  its  forecasting  storms  to  be. 

Used  to  see  where  weather  will  be  for  flights,  however,  not  used  that  much.  I 

Used  operationally  to  get  a  better  feel  for  convective  activity,  l.e.  cells  impacting  NYC,  Newark.  However  product  was  not  easily 
accessible. 

Used  quite  a  bit  -  operationally.  Since  flights  are  generally  short  it  helps  to  see  where  the  weather  will  be  in  an  hour  -  especially 
the  Northeast  corridor.  The  forecast  seems  accurate  based  on  overlaying  printouts  of  the  product  over  radar  one  hour  la 

Used  it  mostly  for  identification  of  convective  weather.  Would  be  more  useful  if  overlaid  on  the  ASD  display.  Like  to  see  big 
picture. 

Used  quite  a  bit  -  whenever  there  was  convective  activity.  Its  one  tool  1  reference. 

Looked  at  it  but  not  used  operationally.  Did  not  see  it  until  the  end  of  the  convective  season  for  4  or  5  days.  Used  other 
information  sources. 

Used  very  little.  Uses  weather  animation  via  Northwest’s  intranet.  Meteorologists  at  Northwest  make  adjustments  and  can  see 
movement  on  product. 
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Response 

Not  exceptional  -  on  a  par. 

NA  -  does  not  zoom  -  wants  ind-zoom.  Does  not  zoom  in  on  Atlanta  Airport.  Delta  really  needs  that  kind  of  zoom. 

No  conclusion  - 1  hour  legs  on  flights.  Uses  WSI  w/loop,  likes  Tops  information  (5). 

Likes  Timelines,  Accuracy  and  Forecast  Plot. 

Viable  tool  as  far  as  convective  product  -  better  definition  than  SIGMET.  Not  easily  accessible.  Will  click  bookmark.  Once  bookmark 
mastered,  got  it  made. 

Uses  WIDs  too  - 1  hour  prod  more  reliable.  NCWF  always  updates,  however,  can’t  get  big  picture  to  look  for  hole.  WIDs  opens  up 
more  to  see  where  you  can  get  through. 

Another  input  device  WIDs  has  bigger  picture.  Can  blow  up  WIDs  to  area  1  am  looking  at. 

Can  also  loop  for  trend  on  WIDs  but  w/  looping  you  can’t  tell  when  it  will  grow  or  dissipate.  Upper  air  charts  lifted  index  but  likes  how 
NCWF  draws  where  position  will  be  Delta  met  dept. 

Prefers  WIDs  because  of  animation  and  zoom.  Met  chart.  NCWF  limited  showing  where  wx  is.  Would  like  to  point  and  click.  Would 
like  to  zoom  in  and  out.  Not  used  enough  to  check  on  reliability. 

Doesn’t  trend  -  no  loop. 

More  current  than  WIDs.  -  Likes  tops  and  speed. 

Likes  WDDs  better  -  look  is  better  -  darker  background  -  uses  more  often.  Would  like  lightning  data,  zoom  feature.  Would  like  GFF  to 
have  better  radar. 

Like  forecast  area  -  good  indication  -  pretty  accurate. 

More  up  to  date  -  updates  faster.  Better  more  accurate,  delineates  storm  better.  Other  data  old.  Intercast  not  as  clear  cut  as  NCWF 
but  easier  to  read.  Chicago  clear  W  and  E  needed  to  click  between  views  to  see  whole  Dicture. 

No  comparison  -  Great  -  Likes  heights  and  movement.  Would  like  simple  extrapolation  based  on  15  min.  animated  increments. 

Resolution  not  as  good  as  WDDs  -  but  not  as  important.  ] 

Likes  Centers. 

The  forecast  is  the  biggest  asset.  Composite  and  N EXRAD  radar  are  used  but  1  can’t  see  all  the  factors.  The  NCWF  allows  that  and  1 
can  get  a  real  world  picture  -  not  just  the  radar.  Used  with  all  other  tools  for  the  big  picture. 

NCWF  was  used  for  the  forecast.  Used  composite  and  NEXRAD  radars  for  detection. 

Takes  one  step  further.  Can  see  looping  on  the  radar  screens.  NCWF  does  the  guessing  for  me. 

Nothing  else  projects.  Can  actually  see  where  weather  will  be  in  1  hour.  The  TAF  doesn’t  show  convection. 

SIGMETs  are  text  only  and  difficult  to  visualize.  If  availbe  it  would  be  used  more  than  other  products  like  the  NWS  briefings  and 
SIGMETs.  Good  tool  to  have  for  a  quick  look  and  in  addition  to  information  sources  already  there. 

Only  tool  we  have  that  forecasts  thunerstorm  movement.  Sometimes  use  the  ATA  website  and  CDM  products.  These  seemed 
accurate. 

Other  products  are  more  accessible  and  easier  to  use.  Like  to  use  the  ASD  display  the  most  which  has  locator  identifiers  making  it 
easier  to  describe  locations  to  the  crew,  l.e.  cells  are  1 31  miles  west  of  Abalene  and  1 30  miles  east  of  Abalene.  With  ASD 

Likes  Northwests  product  weather  animation  for  trending. 

Nothing  compares  to  it  (except  ITWS).  A  unique  product.  Have  been  mentally  drawinq  cvan  forecast  lines  for  years.  A  pre-tactical 
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Did  the  product  add  value  in  performance  of  your  duties  effected  by  convective  weather? 


Response 
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CO 
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Would  add  value  if  accurate  and  had  longer  forecast  -  not  useful  now  for  planning. 

Did  not  add  value. 

Yes. 

Yes.  -  gives  close  idea  of  where  Wx  will  be.  Same  as  WIDs  -  Loops  w/  WIDS.  Basic  idea  of  what  is  aoina  but  no  time  frame  1 

Yes.  Added  value.  Very  informational.  All  information  at  once  ie:  tops,  look  at  1  picture  to  aet  information. 

Yes  -  Added  value  -  worked  together  with  everything  else.  Liked  tops  and  movement  (not  on  WIDs)  if  under  20,000  ft.  not  a  factor  if 
over  50,  -  yes.  W/air  mass  TS’s,  product  can’t  keep  up  with  them.  More  useful  with  frontal  line. 

Forecast  doesn’t  help  much  on  long  haul. 

Looks  to  see  if  it  agrees  with  what  is  happening.  Cross  check  -  one  prod  validates  another. 

Does  add  value  -  esp.  terminal  environment.  Plot  within  5-10  min  before  landing. 

Yes.  Projection  is  helpful. 

to 

JZ 
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CO 

Definitely. 

Yes.  | 

Very  usable,  added  value,  speed  and  tops  are  hard  to  read.  i 

Added  value. 

Adds  value.  —  | 

Adds  value 

Yes,  even  if  it  only  helped  once.  As  we  continue  to  use  it.  it  will  add  more  value.  1 

Helped  to  some  degree.  Better  than  radar  picture. 

Yes,  adds  value.  | 

Yes  it  will.  Can  add  value  to  safety  and  airline  economics. 

Yes,  for  decision  making,  go  or  no  go.  Another  tool  with  radar  summary  and  live  radar. 

The  product  has  value  and  is  a  good  visual  display.  Would  like  to  see  the  forecast  projections  overlaid  on  the  ASD 

Adds  value  for  route  planning  and  back-up  information.  | 

The  forecast  adds  the  greatest  value.  Growth  and  decay  is  still  an  issue. 

There  is  added  value.  Better  than  a  static  display.  Will  compare  to  ASD  to  see  if  the  route  was  impacted 
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Did  product  use  save  time? 

Airline  Response 

;lta _ No  -  Another  tool  _ 

-Ita _ Time  saving  not  important  -  What  is  important  is  more  accurate  data.  Need  accurate  information  for  planning  part  of  bigger  picture. 
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Northwest  |  Theoretically,  yes 


What  product  components,  if  any,  did  you  find  most  useful?  Why? 


Response 

Forecast  -  where  web  will  be. 

1  hour  forecast  (but  not  long  enough)  need  animation. 

Wind  Speed. 

Forecast  contour  tops.  | 

Tops  -  need  to  use  more  updates  good. 

Cyan  forecast  useful  -  stands  out  -  away  from  wx.  Can  see.  Wants  to  see  Tops  speed,  (good  info)  annotation  not  covering  wx. 
background  color  good. 

Likes  to  compare  to  looping.  Will  trend  continue.  1 

User  selected  zooms  overview  -  big  picture. 
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Forecast  tops  and  speed  and  motion. 

Used  RUC  12hr.  convective  outlook  product  -  WSI  up  all  of  the  time. 

1  hour  forecast  tops  not  that  important,  like  speed.  Would  like  to  see  color  chanae  if  switch  in  direction  or  soeed  ! 

50  nm  min  forecast.  Likes  integration  of  information.  Doesn’t  like  whole  US  to  update  -  loses  screen  -  would  not  mind  if  area  he  is 
looking  at.  Only  forecast  1  have  graphically  others  are  verbal.  Prefer  not  to  look  at  different  screens.  WDDs  niotum 

1 . .  -  •  V/N 

Forecast  “ ~  | 

User  friendly.  Tops  information  was  not  helpful,  but  it  gave  a  feel  for  intensitv. 

Components  the  product  has  now  are  pood.  Could  enhance  it  with  animation  and  cloud  heiahts  information. 

Forecast  contours.  Would  like  to  correlate  the  Convective  SIGMETs  with  the  product.  I.e.  bv  number. 

All  current  components  are  valuable.  It  could  use  user  selected  options  and  zoom  capability 

It  anticipated  the  cell  location  in  the  forecast  -  nothing  else. 

likes  color  differentiation,  forecast  capability,  intensity  and  ability  to  hone  in  on  smaller  weather  areas. 

Forecast  component  is  most  useful.  Has  no  strong  opinion  on  other  product  components. 

Forecast.  All  other  components  can  be  duplicated.  Good  pictorial  view  with  timely,  continuous  updates.  Ease  of  use  an  issues.  Would 
like  zoom  capability  like  ITWS,  ability  to  toggle  lighting  and  navaids  on  and  off.  iet  structure,  fix  structure  and  overlay 

The  forecast  component  is  something.  H 

Forecast  component  -  but  still  need  greater  accuracy. 

(iraphic  presentation  and  forecast  -  also  ability  of  stating  one  step  ahead  of  thunderstorms  for  planning.  Reliability  in-house  was  the 
biggest  problem. 
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Describe  typical  situations,  if  any,  where  the  product  and/or  its  components  performed  well. 


Response 

NA  -  Usually  use  WSI  didn’t  want  to  bring  it  up  a  lot  of  clicking  not  easily  accessible  -  not  user  friendly.  [ 

NA 

_c 

o 

3 

E 

j  CO 
"O 
CD 

O 

O 

O 

Z 

i 

o 

2 

Projected  move  West  well.  Gets  severe  stuff  can  see  it  plot  along.  Look  at  movement  and  tops. 

No  -  Can’t  think  of  any.  1 

Flight  NY  to  Dallas.  Wx  was  further  back  from  line.  Contours  showed  where  wx  would  be  in  an  hour.  Could  fly  around  it.  Accurate 
portrayal  of  where  it  would  be. 

Can’t  answer  since  using  both  WIDs  and  it  at  same  time  when  used,  it  was  accurate. 
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03 
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Z 

Useful  -  2  days  ago.  4  fits  no  alternate.  Useful  to  reroute  -  if  moving  would  make  a  difference  -  was  able  to  reroute  given  information.  3 
flights  went  below.  1  went  thru  center. 

Not  really. 

Transcon  flight  -  an  idea  where  to  deviate. 

In  Atlanta.  -  Met  report  forecast  T’s.  However,  not  on  display.  Confidence  in  getting  planes  in  time. 

Storms  over  field  checked  movement  pilot  holding  over  fix  saw  window  of  opportunity.  Could  tell  them  storm  moving  off.  Did  not  need  to 
divert  and  refuel. 

Key  hubstations  for  Atlanta,  Birmingham,  Montgomery  and  Chattanooga.  Chicago  -  Denver  Gulf  area.  1 

Helpful  in  frontal  situation  -  Chicago  -  wave  after  wave  of  Ts.  Not  as  helpful  with  air  mass  Ts 

N/A  '  ' 

Best  when  there’s  a  line  of  storms,  l.e.  storms  from  Canada  to  North  Carolina  were  cutting  off  the  northern  routes.  The  NCWF  forecast 
showed  gaps  to  get  planes  through.  Also  helped  in  confirmation  with  the  Command  Center.  1  could  plan  1  hour  in  advance 

In  Cleveland  there  was  a  solid  line  of  thunderstorms  through  Ohio.  Could  see  some  gaps  north  and  south.  Got  some  planes  through 
instead  of  cancelling  flights. 

Forecast  was  accurate  -  showed  thunderstorm  activity  over  the  NY  area  and  Eastern  PA. 

Was  able  to  reroute  planes  through  line  of  thunderstorms  in  NY  state. 

<: 

z 

Convective  weather  developed  northwest  out  of  Cleveland  through  the  Pittsburgh  area.  The  line  moved  south.  Looking  at  the  radar 
trend  was  ok,  but  NCWF  was  more  specific  and  1  could  make  better  routing  decisions. 

N/A  '  “  i 

N/A 

N/A 

Good  for  field  planning,  l.e.  knowing  speed  and  direction  of  weather  and  whether  extra  gas  is  required.  ASD  used  extensively  to  see 
where  planes  can  get  through  -  NCWF  was  limited  in  this  capacity.  Feel  more  comfortable  using  ASD  with  overlaid  routes. 
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Would  product  animation  add  more  value? 


Response 

Yes  -  Animation  would  be  a  necessity  for  history  and  trend.  I 

Yes!!  Without  looping,  useless,  would  like  zooming.  1 

Perhaps 

Yes  -  but  since  it  is  forecasting,  doesn’t  really  need  to.  Would  be  nice.  i 

Yes  -  works  better,  if  could  move  blue  forecast  line  would  help. 

Animation  -  Yes!!  | 

Yes.  How  we  get  movement  now  -  even  if  forecasting  can  relate  to  where  forecasting  to  -  could  be  used  with  frontal  activity. 

Yes. 

Yes  -  helpful  -  give  more  idea  of  dir.  may  be  redundant  with  projection. 

Yes. 

Yes  -  3  hours  back. 

Yes  -  T akes  a  step  further  of  where  wx  will  go.  1 

Yes.  Would  like  ATC  to  use.  Shared  situational  awareness.  1 

Maybe  confusing  -  don’t  know 

Yes  -  animation  -  at  least  option  -  like  to  see  building.  I 

Yes,  where  it  came  from  and  where  its  going.  1 

Yes,  would  be  easier  to  see  the  direction  and  history  -  trending  aspect. 

d 
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sz 

T3 

13 

o 

w 

0 

> 

Animation  would  be  helpful  -  show  trending. 

Yes,  would  help  show  trends  in  the  forecast. 

Yes,  would  enhance  visualization,  especially  if  it  could  be  toggled  on  and  off. 

Yes,  would  also  like  to  see  zoom,  jet  routes,  navaids,  and  an  indication  of  how  long  before  weather  impacts  an  area.  Future  products 
should  be  3  dimensional 

Yes,  for  trending.  ! 

Yes  with  animation  loops  of  30  minutes,  1  hr  and  1  1/2  hrs. 

Yes  with  1/2  hour  to  hour  loops.  ! 

Animation  would  be  helpful. 

Animation  would  be  helpful  for  trend  analysis. 

Yes,  animation  would  be  helpful. 
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Would  a  longer  forecast  time  be  preferable?  -  For  your  needs,  what  would  be  the  ideal  forecast  time? 


Response 

CO 

1— 

3 

o 

_c 

CO 

1 

CO 

a> 

> 

Yes  -  6  hours 

Yes  possibly.  -  Could  be  useful  -  Maybe  2  hours. 

Maybe  OK  for  1  hour  -  if  hold  together  for  1  hour  all  you  need. 

Would  like  to  see  storms  dissipating  2  hours. 

Yes  -  2  or  3  hours  max. 

For  short  haul,  no  (maybe  2  hours),  For  long  -  4  hours. 

Yes.  4,  5  or  6  hours.  Effects  routing.  If  not  there  could  lighten  load.  Would  like  to  know  what  will  develop  or  what  may  develop  for 
planning. 
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sz 
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CM 

8  hours.  Actually  4  hours  would  be  good. 

2  hours  with  same  accuracy  4-6  understood  not  as  accurate. 

Up  to  4  hours. 

Yes  -  At  least  2  hauls.  Forecasts  in  different  colors,  ie.  2  hour  in  green  and  4  hour  red,  etc.. 

May  want  longer  forecast  on  separate  page  -  not  to  compromise  accuracy  of  1  hour  product.  3  hours  or  less.  1 

Need  2  or  3  hour  forecast,  5  would  be  preferable  of  areas  airplanes  can’t  get  through. 

2-3  hour  forecast  wants  growth.  -  At  least  2,  more  better. 

3  hours. 

Yes,  a  4  hour  graphical  outlook  would  be  better. 

Forecast  of  2  to  3  hours.  j 

2  -  3  hours. 

2  hours  would  be  ideal.  1  hour  is  ok  since  we  deal  with  hour  block  times.  2  hours  would  project  into  ground  time. 

Yes,  4-6  hours  and  planning  and  validation  (comparing  forecast  to  actual  weather) 

!  Yes,  2-3  hours,  j 

1  hour  in  inadequate.  Forecast  should  go  out  to  6  hours  given  known  degradation  in  accuracy. 

Longer  forecast  would  be  better  with  an  estimated  accuracy  determinant.  i 

Forecast  should  be  out  to  2  hours.  2  or  more  hours  would  be  good  for  straqetic  decision  making. 

2-3  hr.  forecast  would  be  ideal  (most  flights  are  1  hr.  15  minutes.) 

Would  be  useful  is  the  algorithm  could  predict  risk  factors,  l.e.  risk  of  convection  in  Fla.  -  can  we  get  planes  into  Miami.  Should  provide  a 
standard  forecast  so  that  everyone  is  thinking  the  same  way. 

I  if  we  can  see  2-3  hours  out  of  NY  it  can  make  a  huge  difference.  Can  prepare  for  worst  case  scenarios.  1 

Yes,  2  hours. 
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10.  What  degree  of  accuracy  is  required  for  a  2  or  more  hour  forecast  product?  What  amount  of  forecast  degradation  would  be  acceptable  for  2,  4, 
and  a  6  hour  forecast? 


Response 

Does  not  have  to  be  accurate  -  just  tell  us  what  you  are  thinkinq. 

2-30% 

4-40% 

6  -  No  more  than  40% 

2  -  20% 

4  -  30% 

6  -  40%  no  further. 

2-10% 

4  -  20% 

6  -  30% 

ATC  is  problem. 

2  -  20% 

4  -  20% 

6  -  20% 

Just  use  for  idea,  can’t  give  %  will  accept  accuracy  will  degrade.  Really  likes  it!! 

2  -  50% 

4  -  25% 

6  -  NA  -  does  not  do  long  haul. 

WIDS  animation.  Exceeds  1  hour  on  NCWF.  Would  like  more  q  and  d. 

No  more  than  20%  for  2  hour,  30%  for  4  and  for  6  "wary".  Can  already  project  an  hour  with  trendinq.  ^ 

Prefers  1  hour  on  short  term  with  dead  accuracy.  Would  like  growth  and  decay. 

10-15%  degradation  would  be  acceptable.  Uses  it  with  everything  else. 

20%  -  2 

20%  -  4 

20%  -  6 

Best  you  can  do  will  accept.  1 

25%  -  2  hour 

50%  -  3  hour 
then  25%  increments. 

As  long  as  updating,  not  an  issue.  “| 

Loss  of  accuracy  understood  as  long  as  it  updates  every  hour  whatever  is  better  than  what  1  have  as  long  as  there  is  a  graphic 

If  it  matched  with  the  radar  loop,  would  use  it  to  forecast  along  with  the  4  hour  outlook.  Would  like  50  -  55%  accuracy  for  a  4  hour 
forecast. 
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10.  What  degree  of  accuracy  is  required  for  a  2  or  more  hour  forecast  product?  What  amount  of  forecast  degradation  would  be  acceptable  for  2,  4, 
and  a  6  hour  forecast? 
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